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THE IMPACT OF ACCREDITATION ON 
PUBLIC SERVICE ELECTRIC AND GAS COMPANY 

H, Denis Hanson 



ABSTRACT 

Public Service Electric and Gas Company (PSE&G) 
became a member of the National Academy of Training 
when the last five programs for its Salem Nuclear Genera- 
ting Station received INPO accreditation in November 
of 1935 , In June of 1986, the NRC conducted a 
post-accreditation visit at the PSE&G Nuclear Training 
Center. In March of 1987, the biannual report for 
the first five programs to be accredited was submitted* 
The development and conduct of training programs, based 
on a detailed analysis of position task requirements, 
complemented with plant-specific, performance-based 
evaluation of trainee performance; has had a positive 
impact on station performance and appearance, as well 
as worker attitude and capability* Communication between 
station and Nuclear Training Department management 
is stronger; and a clearer understanding of the roles, 
reponsibilities , and accountabilities of each has 
occurred* Lessons learned, activities relating to 
the maintenance of accreditation, and the seeking of 
accreditation for a newly licensed nuclear plant, Hope 
Creek, are discussed in the paper. 
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INTRODUCTION 



PSE&G had made a significant commitment to 
comprehensive training for all station workers for 
a number of years, dating well before the advent of 
a requirement for INPO accreditation. Six-month long, 
apprentice level training in all plant operating and 
maintenance positions had been occurring since 1970. 
This was largely generic skill training relating to 
both the nuclear and fossil generating stations. 
Contractor provided courses were also being conducted 
in specialized areas for operator license training, 
instrumentation and control (I&C) personnel, etc. As 
a result of the 1980 union negotiations, training at 
the journeyman level was begun for the electrical, 
mechanical, i&C, chemistry and radiation protection 
departments. College credit recommendations were 
available for a number of the courses conducted by 
the Company. The training staff size was increased 
as a result of the Company's commitment to providing 
this additional training. 

The Company contracted for two control room refsr- 
ence simulators, and a 65,000 square foot training 
center in 1980 and 1981. Over 20% of the training 
facility funding was set aside for the expressed purpose 
of purchasing training equipment. in addition to the 
furniture, office equipment and similar requirements, 
this gave us the capability to outfit each of the shops 
and laboratories with an initial inventory of 
plant-specific training equipment. System simulators 
for instrumentation and control, actual count room 
and other chemistry and radiation protection plant 
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equipment f and extensive electrical and mechanical 
maintenance training shops were established and 
incorporated into the training programs. 

MAJOR MILESTONES 



While a great deal of training was being conducted 
to support the staticns^ the training being provided 
had to be significantly redeveloped to meet the perform- 
ance^based criteria for INFO accreditation. To make 
this change^ relatively large modifications were required 
in each discipline* First, a job analysis had to be 
completed in sufficient detail to assure that learning 
objectives and materials were supportive of the 
comprehensive task list. Secondly, contractor provided 
packaged training programs v;ere replaced by the newly 
hired Instructional staff. They developed programs 
aligned to the job analyses so our instructional programs 
specifically related to Salem Nuclear Generating Station* 
A third significant impact was the incorporation of 
a newly acquired control room simulator for Salem into 
the licensed operator ^training programs , as well as 
in support of some training in other areas. 

Finally and somewhat surprisingly, the most diffi- 
cult challenge and change was to overcome the inertia 
of the training staff from its years of providing topic- 
based training on a generic basiSf to actually accepting 
and owning the need to develop and conduct 
plant-specific, performance-based training* This 
required moving themselves out of the classroom and 
into the station procedures and processes, INPO assist 
visits, following initial submission of self -evaluation 
materials, provided a major incentive for the staff 
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to incorporate the new instructional systems ittethodology 
and begin the significant transition towards a 
performance-abased training system. 

The demand for qualifiad personnel throughout the job 
analysis and program development period was critical. 
During this same timeframe, Salem experienced major 
equipment outages and the need to staff up to meet 
the operating and maintenance requirements of the newly 
licensed Salem Unit 2, 

LESSONS LEARNED 

First and foremost in the lessons learned category 
was to develop a reasonably effective job and task 
analysis (JTA) for each discipline. Our analyses were 
conducted prior to the availability of the industry 
analyses later assembled by INPO, We found that to 
do analyses in sufficient depth to determine the require- 
ments for successful task accomplishment, more 
subject-matter expert personnel time was required than 
originally anticipated* We also found that a single^ 
standardized format for a JTA was not equally effective 
in all job areas. Information relating to frequency, 
criticality, and importance to public safety, provided 
little useful information in either determining detailed 
task requirements or assisting in establishing whether 
the task should be trained to. For both situations, 
experienced subject-matter experts provided better 
information and judgment as to what should be included 
in the training programs. We ultimately used 
standardized job task questionnaires and sampling 
interviews in all job areas , but supplemented the mainte-- 
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nance areas with a comprehensive review of inspection 
orders and work orders to assure tasks performed by 
maintenance personnel were properly identified. We 
have subsequently related our task analysis and 
qualification cards to the maintenance procedures used 
at the stations and use a review of inspection orders 
and work orders for job task analysis validation. 

As most American utilities can attest to by now^ 
completing a comprehensive job analysis is a significant 
undertaking^ requiring talented people. The time neces- 
sary to do an effective job can be easily underestimated 
if the training staff and the plant support personnel 
have not previously experienced the commitment required 
to produce such a document. Equally important to the 
completion of an appropriate job and task analysis 
is to assure that effective methods are in place so 
that the analysis can be maintained current. It is 
very important that I need not repeat the entire job 
task analysis process for each discipline regularly. 
To avoid that; I must have confidence that the JTA 
is being actively kept current. 

A second lesson learned is thait it takes a long 
time for an instructional staff to actually internalize 
a change in the training process , so that your programs 
reflect products of a new way of doing business. This 
was true of staff that had been with the Company a 
long time, as well as newly employed instructors. Those 
staff that had been with us had to overcome the 
i:gnificant inertia of doing things the way they had 
been, which in most cases they felt had been successful. 
New staff/ in addition to understanding and incorporating 
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our way of doing business, had the additional burden 
of learning our plant-specific procedures, people, 
and equipment. Again in this area, it is important 
to guard against regressing from a level of quality 
your staff may have achieved. Many staff members are 
more confortable or find it easier to conduct training 
programs in a standard classroom environment. Once 
you have them oriented to conduct as much of their 
training in the plant environment with appropriate 
mockups and models, or in shops and laboratories as 
practical; without continued encouragement and incen- 
tives, you will find some backsliding into doing it 
an easier way. Figure 1 is a taxonomy we include in 
our instructor training program to focus our instruc- 
tional staff's mind-set, that in dealing with learning 
objectives or evaluation, to attempt to conduct the 
training at the highest performance-based level 
practical . 

Another important lesson is the need to assure 
that plant staff and management, as well as, other 
senior management, are reasonably aware of the effort, 
commitment and resources required to establish and 
conduct comprehensive, performance-based, training. 
Many senior managers' view of training is the "one 
week" type human resource program that are non-site 
specific and pretty much taught out of a suitcase by 
one consultant trainer. This image, when translated 
to nuclear training, leads to a very erroneous view 
of the kind of lesson preparation time and research 
required to conduct good plant-specific training based 
on a systematic analysis of jobs, procedures, equipment, 
and operating experience. Additionally, few if any 
senior managers have had any direct project or management 
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responsibility for the construction , testingf operation^ 
and continued upgrade and maintenance of 

plant^ref erenced , control room simulators, A third 
area often encountered is that those senior managers 
with experience in ^ litary training systems with large 
trainee through-^puT; ^ may eKpect trainers to produce 
a military-type qualification program for a system 
that has an annual replacement need of 2 to 4 techni- 
cians. Creatively different approaches are appropriate 
for these small groups when compared to systems where 
hundreds or thousands of trainees are processed on 
an annual basis. 

The final lesson learned, I would mention at this 
point, is that the acquisition of a quality training 
facility and training equipment including simulators, 
should not lead one to assume that the training equipment 
or training facilities will automatically be effechively 
incorporated into training programs . Perhaps the 
outstanding challenge to nuclear training today is 
the effective training use of a plant-referenced control 
room simulator^ including the evaluation of individual 
and tea^.i performance at the completion of training* 

RESULTS 

In November of 1985, Salem Nuclear Generating 
Station became the second plant in the United States 
to receive INPO accreditation for all ten programs. 
In so doingr PSE&G also became the second member of 
the National Academy of Nuclear Training, This succe^is 
reflected a number of achievements. Firsts the realiza- 
tion that the Company ' s continuing commitment to employ 
people at the entry-level in its work force, and provide 
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in-house training and education to allow employees 
to prograss to the top of their job classifications, 
can be met if an eqiial commitment is made providing 
necessary training and education resources and applying 
appropriate standards . 



As a result of the job analyses, training programs 
wei_^ modified significantly? in one program an extension 
of over two months was required, in other training 
programs, training time could be shortened because 
of overemphasis or lack of appropriateness of some 
training material that had been historically provided. 
The Company/Union Agreement has been revised as it 
relates to training for people in the nr jlear department, 
and regular meetings are held by Company and Union 
officials to discuss changes in the nuclear training 
programs. Communication on a continuing basis between 
the station and the training organization at the manage^ 
ment, as wall as individual discipline level has and 
continues to be very effective* 

The most important impact of achieving INPO accredi- 
tation has been the internalization of an effective 
personnel qualification processes . Station supervision 
and employees understand the use of and need for a 
detailed job analysis if the employees are going to 
be able to complete required complex work activities 
in a correct and timely fashion. Within the training 
organization, the understanding and commitment to helping 
employees learn what is needed to accomplish their 
jobs successfully, is now the center of trainers thought, 
rather than each trainer telling people what he/she 
believes is important, or sea stories from their work 
history which oft-times did not produce focused training 
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of value to the trainee. 

PERFORMANCE 

Most importantly the operating performance of 
Salem Generating Station has shown continued improvement 
over the recent past. While there have been a number 
of contributing factors ^ certainly the qualification 
and capability of the people working there is one of,, 
if not, the most important factor. While training 
received from a systemically developed training program 
is not the only variable that accounts for quality 
performance of people, it is certainly an important 
ingredient* Salem Unit 1 set a free world electric 
production record in calendar year 1985 for a single 
generating unit. In 1986, Salem conducted a refueling 
outage in the shortest period ever for a four-loop 
Westinghouse plant. EKternal measures such as SALP 
and INPO Plant Evaluations are improving. Our new 
Hope Creek Generating Station set world records for 
power ascension testing and initial fuel .load among 
all similar BWR's. All three units have begun 1987 
with excellent operating performance. 

The experience and lessons learned from INPO 
accreditation are being applied to our activities as 
we conduct training for our newly licensed Hope Creek 
Generating Station. As we continue to conduct programs 
for our nuclear generating stations, additional focus 
and definition continue to improve the quality of 
training we provide. 
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Figure li Perf ormance^Based Training Taxonomy 
Perform 

Perform similar 
Simulate 

Simulate similar 

Demonstrate 

Walk thru 

Audio visual 

Discussion 

Read-react 

Lecture 



Note I Often a combination of activities is required 
for effective training. For fundamental training and 
to support enabling learning objectives ^ training may 
consist of activities less directly tied to task perfor- 
mance. For primary tasks # training and trainee evalua^ 
tions should be accomplished with activities as near 
the top of the list as practical. 



"TRAINING-RELATED NUI>IARC ACTIVITIES" 
Robert H* Hoiyoak 
NUMARC DESCRIPTION 

NUMARC , Nuclear Utilities Management and Resourcas Committee, wae 
formed in March of 1984 and ia a confederation of 55 utilities that 
function day to day in the nuclear area. It's genesis wae the initi- 
ative by the Nuclear Regulatory Commission staff in the proposed 
SECY-84-76A regulations which the NRC Commissioners were considering, 
SECY-84-76A covered regulations in management and people related areas, 
rules on training, degrees for shift supervisors ^ rules in maintenance 
activities, senior management on shift and fitness for duty* The execn- 
tives of the nuclear industry did not believe that the proposed regu-- 
lations were in the best interest of nuclear safety and reliability, 
and would have a negative Impact on industry self-improvement initia-- 
tives that were underway and actually could have a negative impact on 
nuclear power plart operation. Enactment of SECY--84-76A would have 
had the effect of freezing the current INPO accreditation program, de- 
stroying the dyn^ies of this growing INPO program. Implementing 
SECY-84-76A would have caused a prudent utility industry to wait and 
see what the NRC would mandate instead of continuing activities to 
enhance programs and strive for excellence, and end up meeting only 
minimum standards as specified by law, 

In June of 1984 industry representatives met with the Nuclear Reg- 
ulatory Commission recommending that the Industry be allowed to continue 
the progress being made without new rule making* Commitments were made 
which will be commented on later in this paper and the Commission agreed 
to delay rulemaking in relation to SECY-84-76A in order to see vmat the 
industry could accomplish on its own. 
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NUMARC created a set of foup basic goals as a basis of Industry 
Initiatives, they arei 



Perform integrated revlewa of management and people-^related 
issues, identify problems and initiate solutions. 

Consider NRC goala in areas of Industry initiatives and help 
the NRC avoid Inappropriate action. 

Establish and maintain a regulatory atmosphere that allows 
flexibility for effective management. 

Inform members of Congresa and their staff of the progress that 
has been achieved by the nuclear industry in Improving the 
safety of nualear power plants. 



The NUMARC executive committee makes the final decision on issues 
and consists of senior utility executives from each utility. 

There Is a steerin,^ committee whose function is to organize indus- 
try Initiatives and coordinate all activities including managementwlth 
other industry groups, the NRC, and Congress. 

The steering committee consists of the Presidents of Georgia Power, 
Virginia Power, Portland General Electric, Detroit Edison, and Yankee 
Atomic; the Executive Vice Presidents of Duke Power, APS, and Conaoll^ 
dated Edison; Senior Vice Presidents from Alabma Power, Arkansas Power 
and Light, and Northern States Power; and Vice Presidents from Philadel- 
phia Electric and Commonwealth Edison* 

Working groups are formed as needed to develop positions on speci- 
fic issues* 
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Industry CQmmltroentg Made On Training Thru NUMARC 

The induetry commitment e relating to training are the following r 

1* Accreditation ; That all plants operating as of Deaemher 31 j 198^1 
would be ready for accreditation in two years* Each plant would 
have ten programed making a total of 610 programs that would have 
to be ready for INPO accreditation by December 31, 1986. 

The 10 programe arei 

Non-Lieeneed Operator 
Readtor Operator 
Senior Reactor Operator 
Shift Technical Advieor 
I&C Technician 
Electrical Maintenance 
Mechanical Maintenance 
Radiological Protection 
Chemistry Technician 
Technical Staff and Managers 

2 ^ Control Room Operator Prof eas_Xpnaliem ; 

The industry committed to enhance the prof esslonaliem of control 
room operators. This commitment pai^tlally evolved out of Senator 
Monahan's coments after a visit to a utility control room that the 
reactor operators were indistinguishable from a group of gas sta- 
tion attendants* Part of the enhancement of professionalism was 
for INPO to include senior reactor operators from other utiiitles 
as peer evaluators on each of their plant evaluation teams to as- 
sist in evaluating on-the-job and simulator performance of licensed 
operator activities. The evaluators from other utilities help with 
the effectiveness of the evaluation process, but also create a lev- 
el of professionalism by having senior reactor operators from one 
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plant visit another plsnt and see how they operate. 

3' Improving D lagnogtlc Capabilltlea of Shift Personnel 

Tha industry cQmaitted to Improving the diagnostic capability of 
shift operating person&el. This was to address a concern of the 
NRG that all shift supfcTviaors or even all control room operators 
should have technical cLegrees. The commitment was to upgrade the 
training in engineering fundamentals so that shift parsonnel would 
have the capability to diagnose abnormal situations. 

^ ' Shortage of Qualifi ed P^reonnal 

NUMARC addressed tha problem of the shortage of qualified personnel 
at nuclear utilltiag an^ the exceesive use of conaultants. Each 
utility would develop P^ane to minimize tha use of contracted per-- 
sonnel in permanent posa^tions- 

5 . Fitness for Dut^ 

The industry committal tea having a fitness for duty program In 
place by January 31, 19&5 which covered aberrant behavior, and 
chemical substance abus» at nuclear stations, based on the EEI 
"Guide to Iff ective Pr*u^ and Alcohol Policy Davelopmant " . 



6. License Requallf icat ion 



Qgram 



NUMARC eet up a working ^group to clear up problems with the NRG 
conducted licanae r equal Icat ions , 

7 • Conduct tralnj^g_WQyk_g&ops of SRQs salacted to participata in SRO 
pear e v a_l u a t'o r_ prgg r ^ fl ^ 

The industry committer fclie utilities to supply SRO peer evaluators 
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on each plant evaluation and NTOL assistance team. A program of 
workahop training for these people was comDitted* 

WUMARC Accomplishments in Training 

NUMARC since its inception in 1984 has In conjunction with INPO 
made a remarkable amount of progress in the training area* The major 
accomplishment has been that working thru INPO ^ of meeting the committ- 
ment of preparing 610 programs for accreditation within the nuclear 
industry- We all know how difficult and expensive It was to do this 
commitment within the tight ta^e table, 

Efforts to improve control room professionalism has made good pro- 
gress. These efforts range from the wearing of uniforms to Improvements 
In simulator training techniques and this is an ongoing effort* 

Diagnostic capability was incorporated into the operating accre- 
ditation format with the result that operator Initial and requallfica- 
tlon Includes exposure to diagnostic techniques and fundmentals* 

An example of meeting this requirement Is what we do at the Common- 
wealth Edison Production Training Center during license and requallflea- 
tion training* We teach diagnostics thru all normal and casualty opera- 
tions but Introduce diagnostics as part of control room skills training 
which Includes Team Building and Communications* 

Diagnostics Is structured similar to many available commercially 
availablek programs^ such as Kepner Traego j BPI or Almo and others* 
Specific classroom exercises introduce diagnostics with made up seenM*- 
ios with forced data that bring the students to a common conclusion * 
Then actual event scenarios using strip charts from recorders, questions 
on a group basis that lead to conclusions » This is augmented with diag- 
nosis on the simulator in real tima* This demonstrates to the operator 
that the designer organized the plant systems with problem analysis in 
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mind and that logic typa aystama do much of the work for you. This 
could be defined as a "claaB A", diagnoBtlc where problems are aelf 
identified by the plant ayBtemH . We work on the, "claaa B" , diagnoBtlcs 
Where the operator has to analyze and work out the problem. One Inter- 
estlng exercise we do la to take an oparating procedure and tranalate it 
line by line Into diagnostic Engliah. Thia acta to clarify many thlngB 
for the operator. 

Utilities have committed to support NUMARC In reducing the exoea- 
alve use of conBultantB by filling permanent posltlonB with utility 



staff. 



Some of the atepa taken include: 

1. Establish a human reBource management aystem covering personnel 
activities relevant to nuclear atafflng. 

2. Support and encourage efforts of area educational inatltutions 
to InereaBB avallabili:y of local peraonnel with solid baaic 
training at the entry level . 

3. Work with area educational inatitutlons to develop continuing 
education programs for utility employeB that are directly rele- 
vant to nuclear staff qualifications and Job progreaa . 

The industry committed toa fltneaa for duty program and this pro- 
gram la in place. 

Again I would like to use Commonwealth as an example. We started 
our first fltnesB for duty program in 198if with a program entitled 
"Awareness" which covered aberrant behavior, alcohol abuse, drug abuse 
and the Edison Bnployee Assistance Program. The next program called 
"Preparednesa" about eighteen months later covered the same areas with 
the addition of atress control and anger management and control. "The 
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current program is being put together and is a reiteration of the pre- 
vioue programs with stress on Edison's experience and policy in the area 
of drug and alcohol abuse. 

Another effort at the request of the utility industry has been a 
coordinated effort to work with the NRC in the area of llcenee requali- 
fication exams ^ Coneiderable concern existed relating to the NRC ap- 
proach to their administration of license requali^?icatlon exMS on a 
random basis. The industry in general considered the NRC intrusion into 
an area, that the utilities considered adequate ^ disruptive and the 
tests in some cases unfair* 

The committee under John Griffin of Arkansas Power has worked with 
the NRC to set up a pilot program where the NRC would be involved in 
overseeing a utility conducted reguallf Icat ion exam^ To date this ap- 
proach has mixed reviews and the pilot program has been extended until 
the end of 1987. 

The bottom line is that NUMARC since its inception since i98A has 
had a major Impact on training. By working with the Institute of Nu- 
clear Power It has moved the nuclear utility training systems ahead In 
time, doing In two years what would have taken a lot more time. The 
existence of a central directorate with the Industry INPO organisation 
has allowed Nuclear Utilities to be quickly responsive to the initiative 
of NRC and the public. 

A statistical Impact of the growth of training within the utilities 
in the training area since the advent of NUMARC can be seen by looking 
at the April 1984 and September 1986 reports from INPO "Survey of Nu- 
clear - related Training Activity in U.E. Electric Utilities", By 
January 1986 training space in the nuclear utilities had increased by 
45? over March of 1984. The number of staff In nuclear training ac- 
tivities increased 85% by January 1986 over March of 1984, 
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NUMARC AND TRAINING, THE FUTURE 

NUMARC has several training initiatives before it. The NRC commis-^ 
sioners Zech, Bernthal and Asselstine have pressed NUMARC to improva 
induatriee maintenance activitlee. Part of that improvement v^^ill center 
on maintenance training and factoring industry experience into that 
training . 

In operator requalif icafcion as mentioned, the pilot program with 
the NRC has been extende ^ and a final recommendation will be made to the 
Commission in mid^l987 . The working group will be meeting with staff to 
review the advantages and disadvantages of the pilot program and dlecuss 
ways to improve thie approach* 

NUMARC has modified the view on the check operator to that of Inde-- 
pendent Evaluator . A survey of the Industry last fall attempted to get 
conaeneus on who does the evaluation, who acts on them, and is it an ef^ 
fective process. These results will be discussed with the NRC staff. 

UNPOC 

Now let me discuss the reorganization of the nuclear industry or^ 
ganizations under the leadership of UNPOC, kthe Utility Nuclear Power 
Oversight Committee. 

UNPOC evolved from an ad hoc group formed in April 1979, to coordi- 
nate and oversee the electric utilities* response to the Three Mile 
Island accident. The original committee was responsible for forming the 
Institute of Nuclear Power Operations (INPO), the Nuclear Safety Anal- 
ysis Center at EPRI , and the U.S. Committee for Energy Awareness. 

UNPOC presently includes utility representation from the following 
associations! The American Nuclear Energy C ounc ii (ANEC), the American 
Public Power Asaociation UPPA) , the Atomic Industrial Porum (AlP) , 
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the Edison Electric Institute (EEI), the Electric Power Research In-^ 
stitute (IPRI), the Institute of Nuclear Power Operations ( INFO ) , the 
National Rural llectric Cooperative Asaociate (NRECA) , and the U.S. 
Committee for Energy Awareness (USCEA) . It has served as a forum for 
issues to be considered at the highest executive level for fostering 
coordination of our induatry organizations. The present Chairman is 
Jim O'Connor - chief executive officer at Commonwealth Edison* 

As a result of the UNPOC-sponsored study ^ a report was isaued last 
year entitled^ "Leadership In Achieving Operational Excellence'-, The 
report has three sections one of which called for the establlahment of 
a new, unified nuclear Induatry organization to be the single industry 
voice on regulatory Issues. This is the section I will discuss briefly, 
because of its continuing impact on training. 

The new organization will be responsible for coordinating all mat-- 
tera involvT^*;? industry-wide regulatory policy issues and ont the regu- 
latory aspcc n of operational and technical safety issues affecting the 
Industry ^ 

The new organisation will be free-standing and independently funded 
and will operate under the concept of NUMARC ~ namely, it will be an 
organization run by the 55 nuclear utilities as opposed to by the ven- 
dors * 

However, the new organization will draw on the collective expertise 
of other industry groups such as archltect^englneers , vendors , and other 
supplies and involve those groups In the regulatory Issues to be ad- 
dressed ^ 

The new organization will be governed by a board of directors com- 
posed of senior executives from each utility that is building or opera- 
ting a nuclear plant. It will be very similar to the current NUMARC 
executive group* The industry -s position on regulatory iaaues will 
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require an affirmative vote by at least 80 percent of tne ffiembera auch 
ae NUKAHC. The new organization will have a permanent afiaff located 
in Wasiiington, D.C. which would have reeponsibillty foi^ the day^to-^day 
interface with the NRG, 

Now, let me give you a quick, short course on the reorganization, 

Tt%m utility Nuclear Power Oversight Committee (UNPOC) will be re- 
reamed- its board will be exyanded to include more chief executive of- 
rdcera of utilities. It will be permanently headquartered and staffed 
required. Under this new constitution, UNTOC will assume responsi- 
bility far strategic planning, coordination, international cooperation , 
a^d the policy leaderehip of the nuclear utility Industry. This will 
exiable tiie overelght comittee to coordinate and oversee directly the 
o^gani^a^tional efforts that are now carried out by AIF, ANEC, NUMARC 
arid USCEA. The American Nuclear Energy Council (ANEC) which was formed 
ixa 1975 and is the governmental affairs arm of the utility, the Atomic 
Industry Forum which was formed in 1953 and handles much of the public 
affairs and technical interface with the NRC and the U,S, Committee on 
Energy Awareness which was formed in 1983 and has responsibility for 
public information on nuclear power, will undergo a structural realign^ 
ment to concentrate their collective resources and capabilities in three 
basic BL^mB.B * 

1 * Regaaaatlon and technical support ; 

2* Communication 5 education and technical services; and 
3. Gov^oi-nmant affairs, 

Eacli of these three reconstituted assoclationB will be organized 
with Its own board of directors, president and full-tlnie staff* 



Thl^ reorganization will be completed and implemented by July 1, 
i98T . 
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Among other considerations ^ this reorganization of the nuclear 
utility industry is expectedl to include a merger of the USCEA 
and AlP minus Alp's regulatory and technical support functions 
which will merge with the functions of NUMARC ^ ANEC viii continue 
to carry out the government affairs functions. 

As called for in the UNPOC report, this new, unified industry 
organization will attempt to interface with the NRC in a nonadver- 
sarial manner. 
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TRAINING AND PLANT PERFQW^CEj^^ ITgATEGIC Jl^^^^^^ 

GPU NUCLEAR CORPORATION 
Dr- . Richard P , Coe 
Directoi^ Training and Educ^ticfi 



^ ABSTRACT 

To no one* s surprises U.S. nuclear utilitte and tMr 
training groups are on a colTision course* h cour% 5% is 
bringing dangerously c~Iose the potentlanj^ opposing ^Vorces 
of the industry 's coninif " tment to excellence ariJthe pub flic's 
relentless pressure for^ cost effectiveness, M opera^jicfiii 
costs continue to esca ^ late^ the cost of replacenient power 
adds to the mounting pr^essure on the nucl ear option. 

The indiistry as a whol €5 and specifically 5Pi fluclearT r ^ Is 
refocusing Its attenti on on perfo^Fnance inclicators, Tliis 
standardized assessinen^^ of plant operatioriil perfori^mQfice 
surfacai numerous e^^amples of hov^ p^rforinance- —based 
training positively iirsipacts plant perfov^rn^nce, Mum'i^erts 
examples of high dolla^r^ savings rang? from mm redu \jction 
programi to reducing p arsonnel reni exposure^. The d»Seeper 
We look the more we find that training is maki Ing a 
difference. The questi on now Is^ how long Crafiwe can^^tffiye 
to afford the ever fn» creasing demands of -ttio purstifrit of 
excellence. 

Early in 1985s the Tr^-lning & Education De^atent ar^ m 
Nuclear proactlvely be^gan Its strategic pi artnlfi^ effort rt in 
order to address the i ncreasing industry initiatives ^ While 
facing flat or reduc^ed commltnients of Resources, Ik 
Training Strategic Plarai addresses detailed f\m for eacfi 
of the foil owing areas: 

'Curriculum PlanMning 
'Program Developr^nent 

■Training & Educ^^tion Organizational S-tructure 
'Training & Educ^ation Administrative Procedures 
° Training Advisowy Structure and Prtoritjprocesi ,5 
' Financial Strat^sgies 

All of the above strat&^gies are designed to mm tra^ iriinj 
effectiveness. With thne nuclear option Undersuch %T t^mi 
public scrutiny 5 it 1s In the best interest of all cn-f the 
nuclear utilities to assure the most eo^t effeoctiv^ 
approach to successfLml operation whil^ acHeving our 
standards of excellence 
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COST VERSUS THE PURSUIT OF EXCELLENCE: 
COMPATIBLE OR INCOfPATIBLE? 

To no one's surprise, U.S. nuclear utnities and their training groups 
are on a conision course. The course is bringing dangerously close 
the potentially opposing forces of the industry's commitment to 
excellence and the public's relentless pressure for cost 
effectiveness. As operational costs continue to escalate, the cost of 
replacement power only adds to the mounting pressure on the nuclear 
option. 

The industry as a whole, including GPU Nuclear, is refocusing its 
attentior on performance indicators. This standardized assessment of 
plant operational performance surfaces many examples of how 
performance-based training positively impacts plant performance. 
Examples of high dollar savings range from scram reduction programs to 
reducing personnel rem exposures. The deeper we look the more we find 
that training is making a difference. 

Led by utility initiatives in the form of the Nuclear Utility 
Management and Resources Committee (NUMARC) and the recently formed 
Utility Nuclear Power Oversight Committee (UNPOC), a standardized 
assessment process will be applied to plant operational performance. 
This process uses nine measurable performance indicators. Plants 
report to INPO in terms of these indicators and based on their 
performance, nuclear units will be rated from "excellent" to "requires 
special attention and assistance." 

These indicators include: 

° Number of significant events per unit » Forced outage rate 

' Number of unplanned automatic scrams = Lost time accident rate 

" Personnel exceeding 5 REM at one facility ° Thermal exposure 

° Collective radiation exposure per unit ° Equipment availability 
Low-level solid radioactive waste per unit 

Since fully implementing our Training Systems Development (TSD) 
process, we have identified numerous instances of training having a 
positive and measurable impact on plant performance. Plant shutdowns 
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were avoided, personnel exposures were significantly reduced and 
costly Cill-outs were avoided resulting In large visible dollar 
savings. Some of the examples are: 

At Oyster Creek, plant scrams had been experienced due to 
Nuclear Instrumentation (NI) interlocks with the Reactor 
Protection System (RPS)* Training was designed and implemented 
on the NI and RPS interlocks both in the classroom and on the 
simulator. Since this training has been implemented no new 
scram incidents have occurred. THIS SAVES S400s000 PER DAY FOR 
REPLACEMENT POWER AND ADDITIONAL CALL OUT COSTS. 

^ At Oyster Creek, operation of the Turbine Control system had 
been a source of recurring problems, A vendor simulator^ which 
had been used as the simulator for operator training^ used a 
system which was very different from Oyster Creek's, Another 
simulator v/ith a turbine control system more like the one at 
Oyster Creek was evaluated and selected. Since switching over 
to this simulator^ the company has noted a significant 
Improvement in operator performance on the turbine control 
system, 

^ At Three Mile Island (TMI), operational upsets and problems had 
been induced by failures and transients in power supplies and 
Instrument input signals to the Integrated Control System 
(ICS). This is common in BAW plants. As a result of training 
designed and implemented by the operator training section, two 
of the newest operators were able to properly respond and manage 
an ICS power failure while the unit was operating at 100% 
power. The plant was able to stay on line at the 100% .power 
level as a direct result of their actions, THIS SAVED $400,000 
PER DAY IN REPLACEMENT POWER AND ADDITIONAL CALL OUT COSTS. 

At TMI, severe reactor imhalance and soluble boron/rod position 
control problems, as experienced at othe:- similar plants , were 
avoided during startup. This was directly related to the 
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awareness and specifTc pr-e-startUfc training v^toTch was designed 
to manage this problem. 

^ At Oj^ster Creek the pre-dr^well %ntrj orientation training was 
directly responsible for reducing t^e accumul ated dose to QA 
Inspectors by a factor of two. The training consisted of a 
short videotape which showed work sreaiin the t-irymll model and 
stm photographs of the mod^l- This Approach reduced 
Inspection time and improved work pialng. r-he job had been 
estimated at 32 REM and actual total exposure te-zas only 14 REM. 
THIS IS A 56% REDUCTION FROM THE EStlM^TED DOSE . 
It IS clear from these examples that tramtng is mUi ng a difference 
using our Training Systems Development (TsD) approach we are confident 
that many more innovative approaches to training p roblems will be 
developed as problems 1n the plants ariSe- The question now is, how 
long can we afford to pursue the eveb tmreasing demands of the 
pursuit of excellence without training bfecMng too fcf»g or too costly 
to afford? 



A POSSIBLE SOLUTJOH 

The answer lies In how Training Managenieht plans ^or the future. 

Early in 1985, the Training S Educatioh Dipartment at GPU Nuclear 

proactively began Its strategic planning Pffort In o»~der to address 

the ever increasing industry ihitiatives Whils facing -flat or reduced 
commitments of resources. The Training Strategic Plan addresses 
detailed plans for each of the following abeas: 

° Curriculum Planning 
" Program Development 

Training 4 Education Organizational St>"ucture 

Training & Education Administrative PrOceJyres 
" Training Advisory Structure and Priobit^ Process 
° Financial Strategies 
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Curriculum Planning 

Our curriculum architecture organizes various training needs Into a 
logical sequence of programs or modules. It provides a blueprint to 
help and assign priorities In developing and maintaining training 
progranis. Prior to embarking on the curriculum design process, a 
detailed project plan is developed for each curriculum project which 
includes estimates of the cost to develop the curriculum and a 
statement of the business benefits to be gained. Project proposals 
are prepared for review and priority setting by the advisory structure. 

Program Developinent 

The prcgram development process ^ as spelled out in our Program 
Development Manual (PDM), is the means for creating training programs 
that directly support job performance. Training has received 
corporate-wide support endorsing the use of this process In all 
areas. Specifically In Training we have: 

^ Established program development resource levels to enable high 

quality program development 
^ Implemented a Curriculum Coordination capability among the three 

sites 

- Implemented a project planning and budgeting system for program 
development 

- Further strengthened the linkage with user organisations 
regarding content by a more consistent use of the Technical 
Content Review groups 

" Implemented an expanded training advisory structure to assure 
proper attention and a corporate-wide consistency 

^ Developed and Instituted a process to augment our Instructor 
staff with plant personnel on special assignment. 
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Traintng and Education Organizatfon Structure 
A revised organization structure 1s proposed for the Training and 
Education Department to more efficiently and effectively carry out Its 
mission and roles. 

Specifically, we have: 

Created new program development capability at each location 
Rerroved some program development responsibilities from the 
delivery groups 

^ Changed some of the mission and roles of the Educational 
Development Department 

^ Added more responsibilities to the training administrative 
support group 

Provided expertise to help develop training. 
Consolidated certain functions 

Instituted rotational assignments to better distribute and 
develop expertise 



Training and Education Adnlnistratlye Procedures 
Several administrative systems have been Identified which make major 
contributions to improvements In both efficiency and effectiveness of 
training* Some of these systems are ther 

Establishment of a uniform Training and Education Data System to 
measure overall training results and costs 

Revision of the Training and Education Budget to separate 
program development from program delivery for priorities and 
budget 

Establishment of a uniform registration and scheduling system 
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Advisory Structure and Priority Process 

In addition to the existing Corporate Training Advisory Council ^ an 
expanded advisory structure is planned which will incorporate a new 
priority-planning process. The roles of the overall advisory 
structure will be toi 

Provide general guidance and direction to the training function 
° Identify needed training programs 

Assure that training Is meeting overall organizational needs 
^ Provide assistance 1n the identification of additional resources 

Assist in setting priorities for training programs 

While there are good formal and informal linkages between training and 
line management at all levels, there Is currently no corporate-wide 
mechanism for setting priorities* 

The Proposed GPUN Training and Education Advisory Structure is shown 
on the chart below* 



GPUN Training Advisory Structure 



GPUN 



Training Advisory Council j 



(Senior Executives) 



Corporate | 
Training Advisory j 
Committee I 



OC Training 
Advisory 
Committee 



TMI Traini ng 
Advisory 
Committee 



Management Level I 



lanagement Level | | Management Level 



4 0 
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Financial Strategies 

Through January of 1986, all of the Traintng and Education activities 
had been financed through a centralized Training and Education 
budget. The line organizations (because of numerous eKternal 
Initiatives) have been placing increasing pressure on this budget. In 
the future, various mechanisms will be established to further share 
the burden of training costs betv/een Training and Education and the 
users. Beginning in 1987r 

Any additional initiatives requested by the user department must 
be funded by the requesting department. 

Working with the Advisory Council, a phased^in approach will be 
developed for sharing development and delivery costs, 
particularly for discretionary programs. 

Training developed and used a charge back system for selected 
programs. 

SUMMARY 

All of the above strategies are designed to assure training 
effectiveness. Our intent Is to reduce the amount of time in training 
without affecting quality, to provide the necessary blend of internal 
expertise and staffing levels, and to reduce the overall cost of 
training. In addition, we will be establishing a priority process for 
training and thus be able to handle new initiatives. 

I do not think It Is a q»f ;tion v;hether or not to do strategic 
planning, but that present 6 ; future events will dictate to us how 
much we will have to do. What I have tried to share with you is a 
brief overview of the efforts of GPU Nuclear-s Training Management who 
are committed to the ongoing needs of their nuclear sites and the 
industry-=wide effort toward the pursuit of excellence. Any plans for 
the future v^lll obviously have to remain dynamic and flexible. 

Training professionals are going to have to constantly challenge 
themselves in order to assure this crucial ongoing support of the 
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nuclear option. We will also have to be constantly aware of the need 
to rapidly accorranodate new Initiatives, Regulators, public advocates^ 
a skeptical public - indeed - the Industry itself will not allow us to 
adopt goals of status quo . I'm confident that our efforts at GPU 
Nuclear have put in motion a process that will assure our ability to 
successfully meet these future challenges. 

GPU Nuclear's aim is to place TMI-1 and Oyster Creek in the front 
ranks of the U*S, operating nuclear plants. With the nuclear option 
under such strong public scrutiny^ It is in the best interest of all 
of the nuclear utilities to assure the most cost effective approach to 
successful operation while achieving our standards of excellence. 
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NUCLEAR POWER PLMT TRAINING 
FROM "THE INSURERS VIEWPOINT 

John Ap Honey 



ABSTRACT 

American Nuclear Insurers and Mutual Atomic Energy 
Liability Underwriters (ANI/MAELU) provides the 
liability insurance policy that responds to the Price 
Anderson Act. The policy covers liability claims 
brought by the ganaral public and third party radiation 
workers resulting from operation of a nuclear facility* 
We all recogniEa tha Important role training plays in 
the continuad safe operation of nuclear power plants. 
Training also plays a key role in liability claims 
evaluauion and thair dafense* This paper deals with 
the importance of training programs in reducing the 
number of liability claims and rhe defense of those 
claims* ANI/MAELU has some unique concerns in that 
regard * 



American Nuclear Insurers and Mutual Atomic Energy 1, lability 
Underwriters (ANI/MAELU) are insurance pools that represent 
approximately 250 member insurance companies that provide liability and 
property insurance for the nuclear industry. The staff of the Nuclear 
Engineering Department at ANi/tlAELU, are responsible for nuclear 
liability risk assessment and reduction for the insurance Pools, These 
risk reduction efforts are designed to both reduce the number of claims 
and to place the Insurance Pools in a better position to evaluate 
claims and* if necessary j to provide the just defense of the claims. 
Nuclear power plant training plays an Jmportant role in this risk 
reduction effort. 

Nuclear liability insurance is provided by the Pools for many 
types of facilities including power reactors, fuel fabricators , 
research reactors * waste burial sites ^ nuclear laundry facilities and 
some manufacturers of products using nuclear materials. The insurance 
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policy at power reactors meets the financial protection requirements of 
the Price-Anderson Act. Among other things, the nuclear liability 
policies issued by the Pools Include an omnibus insured provision which 
acta to channel all claims for nuclear liability coverages to the Pool 
Issued insurance policy for all activities undertaken at the facility 
by all parties. The ANI/MAELU nuclear liability policy therefore 
responds to claims brought by the general public or third party workers 
alleging bodily injury or property damage resulting from operation of 
the nuclear facility. 

Why is the ANI/MAELU Nuclear Engineering Department concerned with 
nuclear liability risk reduction? One could argue that there are many 
regulations governing the operation of nuclear facilities designed to 
protect the general public. If the nuclear facility operates within 
these regulations, the public is protected by definition and therefore 
they have no grounds on which to file a claim. Unfortunately, that 
argument is not supported by reality. 

Since the inception of the Pools in 1957 there have been 122 
incidents that resulted In claims. Thirty-nine of those incidents 
occurred prior to January 1, 1979. Over two thirds of the incidents 
resulting in claims have occurred in the last seven years of the 
insurance Pools' thirty years of existence. I propose there are three 
primary factors that are responsible for that recent upsurge in claims. 
These are notoriety, the age of the industry and the number of workers 
in the industry. 

Notoriety Is the factor which most precisely defines the "turning 
point" in claim trends, since that point coincides within a few months 
with the Silkwood trial and the events at Three Mile Island. In 
addition to these well known events, of course, are such controversies 
as the Mancuso Study at Hanford; the Najerlan study of Portsmouth Naval 
Shipyard; the Bross report on the Tri-state Data; the "SMOKEY" 
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incident and the much publicised pronouncements of anti-nuclear 
activists. Although it cannot be precisely established to the 
satisfaction of all^ a prudent person would consider such publicity 
having adverse impact on claim trends. 



The age of the nuclear industry is a \,\ . ; ^ chat becomes more 
important each year owing to the increaslnf ..ge of the work force. The 
older a worker becomes, the higher the probability for a naturally 
occurring cancer - and it is a reasonable assumption that some in the 
nuclear workforce who develop cancer will attempt to make a claim 
against the nuclear industry ^ regardless of the cause of his cancer. 
Perhaps* the same may be true of members of the public living near 
nuclear facilities. 



Finally t the increasing number of facilities and larger number of 
workers per facility wills as a matter of probability, increase the 
number of claims in the future. The degree to which this will be true 
depends largely on the effectiveness of the first two factors in 
producing new claims. In fact, there is a very strong relationship 
among these three factors* The importance of these three lies in the 
fact that they can only become worse from the insurance viewpoint I the 
bad publicity is already occurring (and may well get worse); more and 
more workers will develop natural cancers as the Industry -ages and the 
number of workers will continue to increase for many years. 

In my opinion, many of the claims to date have questionable 
technical validity, 1 am sure most of us assembled here woulu not 
expect radiation exposures of a few milllrem or plant discharges well 
below regulatory limits to result in claims. To the contrary, they 
have resulted in claims, the more interesting of which are illustrated 
in Table 1. 



Unfortunately, it is not the responsibility of engineers or 
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scientists to pass judgement on the validity of present or future suits 
for bodily injury or property dainage. Juries made up of members of the 
general public make chose judgements. They tend to be more sympathetic 
to the claimants than to the nuclear industry* 

As I mentioned earlier, there are things we can do to Improve the 
situation. We can strive to reduce the number of incidents that can 
lead to claims by Improving the performance of power plants. Also we 
can put ourselves in a better position to evaluate and defend claims. 
1 am sure that we all appreciate the role of training in improving the 
performance and hence the safety of nuclear power plants so I have 
chosen to limit my comments on that aspect. The combination of the 
INPO Training Accreditation Program and the adoption of unit specific 
simulators should contribute to an increase in the quality of training 
received by plant operators and concurrently decrease even more the 
fraquency of incidents that might give rise to nuclear liability 
claims ^ 



I would like to concentrate on the importance of training in claim 
defense. One of the major allegations of claimants is that they ware 
not properly warned of the radiation hazard or properly instructed on 
how to protect themselves from it. Good training programs not only aid 
in reducing the number of incidents that may lead to claims but aid in 
the defense of claims that result. It is important to be able to 
demonstrate worker competence through a good training program and be 
able to show that the worker had maintained his/her qualifications. If 
an Incident were to occur at a time when an operator 'a qualifications 
had lapsed or after he/she had demonstrated a lack of qualifications 
then claims associated with that incident may prove difficult to 
defend. 



Allegations are often made that an employer did not care about the 
1th and welfare of his employees. An eKcellent and well documented 
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training program provides sound evidence Lo refute such an allegation, 

Atll/MAELU inspects all of the commercial nuclear power plants in 
the country. One of the areas we review are training programs because 
of their importance in claim defense and evaluation* I would like to 
devote the remainder of my talk to some of the specifics we look for* 
As I mentioned earlier s there are two areas of concern. These are 
claims brought by nuclear workers alleging personal Injury from 
radiation exposure and claims brought by the general public alleging 
personal Injury and/or property damage from the release of 
radionuclides. The first concern is partially addressed through 
General Employee Training which provides sufficient warning of hazards 
and adequate training in safe procedures for dealing with those 
haEards, It is important to be able to reconstruct the training each 
worker received and prove that he received it and was given the 
opportunity to be effectively trained. Therefore we recomiiiend that the 
syllabus or lesson plan containing topical outlines be retained. Since 
much of the instruction may be presented orally, an exact transcript of 
the material is often not possible. These documents should show the 
effective period of use. They should be treated as controlled material 
with retention of the master copy of each revision. 

Appropriate passing grades should be established as a requirement 
for the successful completion of the radiation protection portion of 
the GET examination. Where the radiation protection examination is 
broken down into separate sections, each section should be passed. 
Persons requiring re-examination should not be allowed to take the same 
exam twice. Examinations covering radiation protection subjects should 
be reviewed formally with students to clear up any misconceptions of 
rules* concepts and practices* The review should be documented by the 
trainee's signature which acknowledges the revievy Miid the opportunity 
to ask questions. 
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GET ref rasher training should be conducced annually. Examinacions 
which measure proficiency In radiation protection may bo used in lieu 
of the refresher training. However, refreaher training should be 
conducted biennially on changes, problems, and new policies. If an 
individual fails the annual examination, unescorted access should be 
withdrawn immediately. Such individuals should receive the complete 
initial GET-radiatlon protection training program and pass another 
examination prior to reinstatement of their unescorted accass. If an 
Individual was allowed unescorted access after he had failed an 
examination on the subject and then allegad injury due to radiation, 
it may be difficult to establish that a reasonable effort had been made 
to protect him. 



Any person touring the facility on an infrequent basis is a casual 
visitor, and even though escorted, presents a concern to the liability 
insurance Pools. It does not seem reasonable to devote the time end 
effort of GET training but something should be done to warn visitors of 
the radiation hazard and assure their protection while on site. 
Therefore It is prudent to instruct visitors in the following^ 

a. Warning of the potential hazards associated with entering 
radiation and potentially contaminated plant areas. 

b. Radiation zones and signs used to warn of the potential 
hazards . 

c. Proper use of protective clothing (if required for entry). 

d. Proper use of step-off pads (if required). 

e. Escorted individuals should not be separated from escorts. 
If an emergency requires that an escort leave, the escort 
should give instructions to the escorted Individuals to 
proceed to the nearest assembly point. 
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Also escorts of casual visitors should have sufficient training to 
assure that the casual visitor is protected in all areas of the 
facility to be visited* 



Another concarn arises in relation to female employees. Due to 
the increased susceptibility of unborn children to radiationi feniale 
eraployeeSj their supervisors, and co-workers should receive instruction 
concerning prenatal radiation exposure. Receipt of this training 
should be documented. 

Our second area of concern is claims brought by the general public 
alleging personal injury and/or property damage from the release of 
radionuclides. The frequency and amount of releases during normal 
operation and especially upset conditions are strongly related to the 
proficiency of the operators. Training obviously plays a significant 
role in developing and maintaining that proficiency. We are very 
pleased with the progress made in this decade with regard to licensed 
and non-'licensed operator training. The increased use of plant 
specific simulators, the INFO Accreditation Program and training based 
on Job task analyses are very significant. However, there is one issue 
dealing with requalif ication examinations that deserves note. if an 
operator fails a requalif ication examination, we have recommended that 
he should be immediately rem.oved from the watch bill until he can again 
demonstrate his qualification* We have taken this position for two 
reasons. His knowledge of the facility and its operating procedures 
has lecome less than desired* He could cause or contribute to an 
Incident that directly or indirectly results in increased releases and 
a claim by a member of the general public. Such a claim could allege 
negligence on the part of the utility for allowing the operator to 
continue operating after he has demonstrated a deficiency in his 
qualifications . 
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In summary, the nuclear Insurance Pools have aome unique concerns 
that we are pursuing regarding third purty nuclear liability claims and 
the evaluatlpn and possible defense of those claims. Power plant 
training programs and their documentation play an important role in the 
Pools risk reduction effort. Wa are pleased with the Improvements made 
m training programs at nuclear power plants and will continue to work 
with the nuclear industry in implementing training program 
improvements . 
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CatagQry 



Chronic Exposure 



Negliganca in Operation 



EKposure of Public 
ALABA Program 

Ganeral Employaa Traininf 
Due Cara 

Technical Overaxposure 



Allegation 
Padiatlon Exposure 

Failure to follow proceduras 
Negligence in Maintenance* 
Oparation, and Inspactlon 

Plant Releases 

Negligence in Controlling 
Exposure 

Failure to Train EmplDyeas 
Failure to Warn of Health hasard 

Conscious indifferanca to welfare 
and rights of worker 

EKceeding Regulatory Limits 
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TRAINING BENEFITS, FROM NSSS OWNERS GROUP PARTICIPATION 



R. C. Nine 
J. E* Jones 
C. Ruzich 



ABSTRACT 



Even though the event at Three Mile Island was a bleak 
mQment in the history of nuclear powers many advances in 
the nuclear industry have evolved as a result. One such 
advancement involves the establishment of NSSS Vendor 
Owners Groups. These groups were organized on a voluntary 
basis with nearly all utilities participating. The main 
purpose was to achieve mutual benefit^ both technical and 
financial s through joint engineering and plant operation 
programs. This paper focuses on the Westinghouse Owners 
Group, which is commonly referred to as the WOG^ and how it 
has benefited and could further benefit utilTty training. 
The paper consists of three sections. The first section 
provides an overview of the WOG structure and how it func- 
tions. The second section focuses on the major accomplish^ 
ments of the WOG with emphasis on the development of the 
Emergency Response Guidelines (ERGs). The third section 
provides some recommendations as to how utility training 
departments can better utilize their owners groups. 



WESTINGHOUSE OWNERS GROUP STRUCTURE AND FUNCTIONS 



The Westinghouse Owners Group (WOG) was founded in June^ 
1980, Presently 100 percent of the domestic Westinghouse plants 
(totaling twenty^five utilities either operating or under 
construction), seven international utility consortiums, and 
Westinghouse are members of the WOG. 
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The WOG consists of utility representatives from which officers, 
committees, and working groups are selected. The structure comprises 
the Steering Committee, technical subcommittees, and working groups as 
shown in Figure 1. The Regulatory Response Group (RRG) and the Issues 
Review Group (IRG) provide rapid response to industry and regulatory 
issues. The Executive Advisory Committee (EAC) assists and supports 
the WOG organization with executive utility management influence and 
advice. The W Planning Board (WPB) provides Westinghouse management 
counsel on technical issues suitable for WOG involvement. 

The WOG functions primarily through the Steering Committee that 
receives input from WOG utility members. Westinghouse, the WPB, the 
RRG, the IRG, and the Nuclear Regulatory Commission (NRC). Input from 
the various groups is carefully reviewed by the Steering Committee. 
If this committee determines that an issue requires further study and 
evaluation, it will assign an appropriate WOG subcommittee to do the 
work. The subcommittee further reviews and evaluates the issue and 
makes recommendations as to how to resolve it. The recommendations 
can include programs that may be funded by the WOG in order to ade- 
quately address the issue. Any program or programs that are endorsed 
by the subcommittee are returned to the Steering Committee for addi- 
tional evaluation and review. If the Steering Committee endorses the 
programs submitted by the subcommittee, then they are presented at the 
next WOG General Session where they are voted upon by individual 
utility representatives. 



A two-thirds majority vote is required to approve any program for 
funding. Only the domestic utility members have voting rights 
(twenty-five votes total); Westinghouse has no voting privileges. 
Given a majority vote, the program is then supported by the full WOG 
membership with few exceptioni. This unanimity of the WOG membership 
IS desirable. It it ensures more active participation from the full 
WOG membership because the program is important and generic to all 
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utility mtmbers. Also, an Important aspect of this process is that 
the WOG represents its entire membership to the NRC on any of the 
approved programs* After a program is passed by the two-thirds vote, 
the Subcommittee that submitted it Is tasked to direct and monitor the 
program to completion* 

The WOG membership financially supports WOG administrative activities 
with an annual stipend for this purpose, i.e., project management. 
The funds for Individually approved programs are allotted at the time 
of approval by the WOG, Each WOG member utility pays a fraction of 
the total cost of the programs based on its number of shares (one or 
one^half) as compared to the total number of shares. The total number 
of shares is equivalent to the total number of utility voting shares 
plus the seven half international shares plus three shares from 
Westinghouie for a total of thirty^one and one^-half shares. A utility 
share can be exempted from paying for a given project if an exemption 
is approved by a two^thirds vote of the WOG membership. 

There are some cases where the WOG also allows the development of 
specific, nongeneric programs that are applicable to less than a 
majority of the membership. These programs are funded only by the 
participating utilities. 

In conclusioni the WOG functions with a deliberate and formal process 
when implementing programs. This process consists of frequent checks 
and balancas in order to maximize the overall benefits of any funded 
programs* It also attempts to use the considerable resources avail- 
able within the member utilities to the greatest extent possible. 
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steering Committei 

The Steering Committee can be considered the spear h&^C of the WOG. 
It consists of the Chairman and Vice Chairman of the WOG, the Chairman 
of the RRGs and the Chairman of each subcommittee. Refer to Figure 2. 
Each member of the Steering Committee is a utility representative 
selected by the WOG membership. The Steering Committee functions as 
a filter in the review of all input. If the input has a generic appli- 
cation to the WOG, then the Steering Committee assigns the most appro- 
priate subcommittee to review it* Most input that is not reasonably 
applicable to the majority of WOG membership is discarded. The 
Steering Committee has the following responsibilities: 

0 Review and/or recommend issues to be presented to the full 
membership of the WOG 

0 Review the results of the technical subcommittee evaluation of 
issues prior to presentation to the full owners group 

0 Recommend policies to be presented to the WOG 

0 Meet prior to a WOG meeting to formulate recommendations 

0 Perform a strategy and planning function that reviews 

potential future issues; provides integration of planning; 
monitors consistency between owners groups, the NRC, and 
industry; and interfaces with other owners groups. Institute 
for Nuclear Power INPO, and NUMARC 

0 Review the establishment of additional subcommittees, and set 
priorities for all work efforts 
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0 Review the estabHshment of technical subcommittees for 

nongeneric issues^ and coordinate and set prioritias of the 
activities of the subcommi ttaes 

0 Determine methods for funding specific efforts requiring 
additional work to envelop all plaiits 

0 Determine technical and financial participation by a new 
member for work previously performed 

0 Report once a year on the status of the WOG activities to the 
EAC 

Groups Providing Input to the Steering Committee 

Many problems and issues experienced by one utility are common to 
other utilities* Utility WOG membars can input their concerns and 
issues to the Steering Committee for review and possible funding by 
the WOG as a generic issue. The WOG provides a cost-effective means 
to address common plant issues and problems^ 

Wastinghouse Electric Corporation also provides input to the Steering 
Committee concerning issues generic to WOG members. Because of 
Westinghouse's involvement with the utility plants* it has a large 
pool of engineering information and expertise that can be very bene~ 
ficial in identifying generic issues that affect all WOG member 
utilities. It also has tri expertise to resolve many of the issues 
funded by the WOG. 

The EAC provides easy access to executive management support from 
member utilities. Acting primarily through their annually elected 
Chairman and Vice Chairman^ this committee has the responsibility of 



DO 



11,5.5 



keeping abreast of the WOG's overall direction and program objectives. 
The committee is also available to assist the RRG and the subcommittees 
in any discussion needed with higher level NRC management or the NRC 
Commissioners. 

The RRG is a quick-response organization within the WOG that provides 
inputs to the Steering Committee regarding regulatory matters. It 
functions to provide the NRC with positions or immediate actions on 
potential generic plant safety concerns* The RRG membership also 
includes the Chairman and Vice Chairman of the EAC who serve as 
advisory members. 

The WPB consists of Westinghouse middle^management personnel* It was 
established in late 1985 with the objective of providing counsel to 
the WOG Steering Committee in the review and prioritization of 
generic issues for potential WOG involvement. Strategic areas 
addressed by the group include improved plant avanabilitys improved 
maintenance practices, performance optimization, and regulatory 
improvements. 

The IRG consists of the WOG Chairman, Vice Chairman, RRG Chairman, 
and the appropriate technical subcommittee Chairman. This group is 
responsible for disposition of industry issues requiring expedient 
resolutions, e.g., the Davis Besse Incident. 

The technical subcommittees provide support to the Steering Committee 
by extensively evaluating issues identified as being generic to the 
WOG members and by monitoring their respective programs. Subcom= 
mittee recommendations are supported by appropriate information and 
documentation. Currently five subcommittees are functioning as part 
of the WOG: Operations Subcommittee, Analysis Subcommittee, Technical 
Specifications subcommittee, the RRG, and the Materials Subcommittee. 
Refer to Figure 1. Special working groups are sometimes included 
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within iubcommittees* The Trip Reduction Assessment Program Working 
Group is such a working group and reports to the Operations Subcommit^ 
tee. The WOG membership is well represented on each of the subcom' 
mittees and working groups. The responsibilities of the subcommittees 
are defined as follows: 

0 Establish technical work to be performed and recommend to 
Steering Committee and WOG appropriate direction and costs 
becausi the expenditures on generic WOG efforts are voted on 
by the full WOG, 

0 Maintain records of subcommittee activitieSj and distribute 
meeting minutes to all subcommittee and WOG members 

0 Monitor and approve costs incurred relative to funding 
authorized* 

The charter has a provision for smaller groups of utilities to func- 
tion under the protection of the full WOG. This provision has been 
utilized occasionally to the benefit of subgroups of WOG member 
utilities faced with particular issues to resolve. Examples of these 
iss^^ include further studies into steam generator tube ruptures, 
tur^^uie governor valve surveillances outside^containment mass energy 
releases, and main steam safety valve discharge capacity* 

PROGRAMS COMPLETED BY WESTINGHOUSE OWNERS GROUP 

Becausi the WOG primarily addresses issues that are generic to the 
member plants, virtually all the money appropriated for programs to 
date has been spent on three major areas of concern. These are generic 
analyses, the Emergency Response Guidelines (ERGs) used by the utility 
to develop plant^specif ic emergency operating procedures (EOPs)^ and 
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improvements to plant technical specifications* Table 1 lists the 
major post-TMI regulatory action items that have been undertaken by 
the WOS since 1980. Table 2 lists the major issues being worked on 
by the WOG. 

The WOG has generated a large amount of technical information from 
its various programs. This information is in the form of periodic 
program reports, WCAPs, and letters. These documents are distributed 
to utility members through WOG meetings, seminars, and mailings 
directly to utility representatives. Reports that are developed from 
each WOG program may contain beneficial information for WOG member 
plants, especially the training department. The utility training 
department can access this information through its WOG Representa^ 
tive who receives a copy of all of the documents generated from WOG 
programs. 

The WOG provides generic information which can easily be evaluated by 
the utility to determine its proper use and implementation in their 
own plant programs. Training Departments that play a role in this 
effort can easily assess the materiars usefulness in improving 
personnel performance in the areas of operation, maintenance^ 
engineering, and regulation. 

The key to effective utilization of the WOG generated materials is 
the utility representative to the WOG. This person is the focal 
point for any communications between the WOG and the utility person 
needing the information. Evaluation and possible implementation of 
WOG materials into training programi may result in many unanswered 
questions. Training personnel should be able to access their WOG 
utility representative with any questions they may have on the 
materials. If the representative is unable to answer the questions^ 
it can be referred to the appropriate subcommittee that developed the 
materials. Many times the question may be answered by direct contact 
with those who performed the work. 
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Table 1, List of Some Issues Undirtaken b^^ 
Westinghouse Owners Group Since 1980 



Opera tional Issues 

!• Development of Emergency Response Guidelines for implement 
tation into plant specific Emergency Operating Procedures 

2, Task analysis of ERGs to identify control room operator tasks 
for use in Control Room Design Reviews 

3. Davis^Besse Incident Evaluation of NUREG findings and 
assessment of their relevance to WOG plants 

4« ERG Training Material Development and Seminars with emphasis 
on Pressuriied Thermal Shocks Steam Generator Tube Rupture^ 
and Loss of Coolant Accidents 

5. NRC ERG Seminar to familiarize NRC examiners with ERG format 
and use 

6. Post Accident Sampling for Core Damage Assessment based on 
radioactive isotopes contained in primary coolant after an 
accident 

7* Detection and Cause of Voiding during RCS Transients and void 
control as per NUREG-0737 

Analysis Issues 



1. Relief and Safety Valve Performance Test to verify reseating 
capability during the relief of steam and water 

2. Pressurized Thermal Shock effects of Small Break Loss of 
Coolant Accident analysis which allowed less restrictions on 
plant operations 

3. Power Operated Relief Valve Failures/Sensitivity Studies which 
satisfy the NLIREG-0737 requirements for PORV failures and 
testing 

4. Auto Trip of Reactor Coolant Pumps during LOCA analysis which 
indicated that an auto RCP trip is not necessary during LOCA 
conditions for Westinghouse NSSS 
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Table 1 (Cont). List of Some Issuas Undertaken 
by Westinghouse Owners Group Since 19B0 



5, Experimental Verification of RCP Seal Integrity Performanct 
with Loss of All AC Power 

6. New SBLOCA Model and Plant Specific Analysis to assure 
continued safe reactor operation after NRR review as per 
NUREG-0737 

Technical Specifications Issues 

1. WOG Technical Specification Optimization Program use 
probabilistic risk methods to assess and revise as appropriate 
outage times and surveillance intervals for reactor protection 
systems and engineering safety features instrumentation system: 

2. Anticipated Transient Without Scram (ATWS) Rule which provides 
a diverse turbine crip from the reactor protection system and 
auxiliary feedwater start in order to limit pressure rise 
during an ATWS including defense against diverse SCRAM to 
address the need for a second reactor trip method 

3. Anticipated Transient Without Scram Mitigation System 
Actuation Circuitry (AMSAC) Design development and submittal 
to the NRC for review and approval 

4. Elimination of Diverse SCRAM Program evaluated the need for a 
second reactor trip method 

5. Reactor Trip Breaker Issues (Maintenance/Qualifications) which 
quantifies a basis for breaker test/maintenance practices 

Materials Issues 

1. Pressurized Thermal Shock screening criteria to show that risk 
of vessel failure is acceptably low with options if criteria 
exceeded 
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Table 1 (Cont). List of Some Issuis Undertaken 
by Westinghouse Owners Group Since 1980 



2* Irradiation Damage Trend Curve provides greater margin below 
PTS screening criteria and increases heatup and cooldown 
margins for plant pressure/temperature curves 

3p Reactor Vessel Beltline Materials Database to determine best 
estimate copper and nickel values for beltline welds 

4. WOG Input for NRC NSSS Bolting ConcernSs Generic Issue 
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Table 2, Recent Issues Undertaken or in Progress 
by the Westinghouse Owners Group 



Operational Issues 

1. Davis Besse Evaluation identifies potential issues from the 
incident in order to enhance the safety and reliability of 
WOG plants 

2. Evaluation of MOV Common Mode Failures During Plant Transients 
Resulting from Improper Switch Settings in order to ensure 
that torque switch settings are proper 

3* Methodology for Elimination of the Spray Additive Tank in 
order to simplify plant design, eliminate hardware, reduce 
maintenance/testing/radiation exposure, and eliminate 
Technical Specification requirements 

4* ERG Maintenance Program systematically obtains, documents, 
evaluates, and reports un feedback from the ERGs 

5, Trip Reduction Assessment Program evaluates plant trips and 
develops software and hardware modifications that will reduce 
the frequency of plant trips 

Analysis Issues 

1. Severe Accident Policy Compliance for Industry Degraded Core 
(IDCOR) which follows severe accident evaluation methodology 
and minimizes possible major plant specific studies 

2. Dropped Rod Protection System which provides an NRC approved 
basis and defense for the elimination or relaxation of the 
negative flux rate trip or turbine runback for plants which 
rely on those features for dropped rod protection 

3. Small Break LOCA (SBLOCA) NUREG 0737 development of SBLOCA 
code and model to assure continued safe operation per 
NUREG-0737 
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Table 2 (Cont), Recent Issues Undertaken or in Progress 
by the Westinghouse Owners Group 



4, RCP Seal Integrity Program analyses for RCP seal condition 
after a loss of all electrical power without any RCP seal 
cooling for WOG plant licensing and safety issues 

Technical Specifications Issues 

1, Technical Specif ication Improvement Program which identifies 
and evaluates problems with Technical Specifications and 
recommends potential solutions to the NRC 

2, Reactor Trip Breaker Maintenance/Surveillance Optimization 
Programs provided technical information on the RTBs with 
emphasis on monitoring and surveillance 

3, WOG Technical Specification Optimization Program uses 
probabilistic risk methods to assess and revise as appropriate 
outage times and surveillance intervals for reactor protection 
systems and engineering safety features instrumentation systems 

4* ATWS Mitigation System Actuation Circuitry (AMSAC) Generic 
Design Program licensed an AMSAC system approved by the NRC 

5. Generic Design of Protection Channel Bypass Circuitry 

Materials Issues 

1* NSSS Primary Boundary Bolting and Closures Instruction Manual 
provides torque specif ications for bol ts/fastenerSs lubricants^ 
and bolt materials for WOG plants 

2, Irradiation Damage Trend Curve Development 

3, Enhancement of Data Base for Irradiated RV Beltline Materials 
based on surveillances capsule^ and test reactor data 

4, Evaluate the Effects of Aging in Cast Stainless Steel on 
Structural Integrity for pump casings and valve bodies in 
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Table 2 (Cont). Recent Issues Undertaken or in Progress 
by the Westinghouse Owners Group 



order to eliminate inservice inspections and thus reduce 

man-rem exposures 
5. UT Inspection of Main Coolant Loop Welds Program which 

provides more meaningful inservice inspections on Cast 

Stainless Steel piping 
5. Flux Thimble Blockage Program identifies causes of blockage 

and provides recommendations to reduce or eliminate it, thus 

reducing utility maintenance costs 

Plant Life Extension concentrates on critical plant components 
evaluation that may prohibit the extension of plant life 
S. Component Cooling Water System Replacement Corrosion 

Inhibitors Evaluation Program which assesses the effectiveness 
of alternate corrosion inhibitors on system components and the 
environment, thus eliminating chromate releases 
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The ERG Development Program is an example of excellent technical 
materials that included training programi programs by the WOGp After 
TMI^ it was detirmined that the emergency procedures were grossly 
inadequate for accidents consisting of multiple failures. The proce^ 
dures tended to address a single event rather than addressing multiple 
plant failures, ThereforSp the operators were locked into a single 
event with actions that may or may not have been appropriate for the 
symptoms in the plant. The actions taken could have even further 
degraded the accident. Because of this, the WOG approved a program 
to upgrade the emergency procedures. The program went through many 
iterations, which ultimately resulted in the Emergency Response Guide-" 
lines. The ERGs are symptom based rather than event based; they have 
the operators take actions based on the most critical plant symptoms. 
If a steam generator tube rupture has occurred coincident with a 
steamline break, than the ERGs would first address the symptoms of the 
steamline break. After the appropriate actions for the steamline 
break have been taken, actions for the steam generator tube rupture 
symptoms are taken. The ERGs provide flexibility in handling multiple 
plant failures in the order of highest priority. 

The Emergency Response Guidelines are based on a generic Westinghouse 
plant. However, it was recognized that the member plants could be 
separated into two categories; the high^pressure injection p]ants and 
the low-pressure injection plants. Generic guidelines were developed 
for both the generic high-pressure and low-pressure plants. 

In order to provide consistency in the writing of the guidelines, a 
Writers Guide was developed. This Writers Guide provided a rigid set 
of rules for writing each guideline consistently. A generic Users 
Guide was also developed so that the guidelines would be used as 
intended by the developers. Any improper use of a guideline could 
result in improper responses and further degradation of plant condi- 
tions. Therefore, consistency in writing and use of the ERGs was a 
priority in the ERG Development Program. 
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With the development of the ERGs, it was recognized that the utility 
writer for the plant-specific EOPs would have to understand the intent 
of guidelines and guideline steps in order to adapt the ERGs to their 
plant* Thereforts extensive background documents were written in order 
to provide the utility writer with sufficient information to adapt the 
ERGs to plant-specific EDPs. An extensive amount of analysis was per- 
formed for the development of the ERGs since the FSAR did not analyze 
for multiple events* All this material was incorporated into the 
background documents. 

Generic issues arose as the ERGs were developed. These issues were 
addressed separately from the background documents* HQwever, the 
generic issues were referenced in the background documents for any 
step or steps that may have needed further amplification. The Writers 
Guide, Users Guide, history of the ERG development, and generic issues 
were incorporated into one volume, the Executive Volume. 

After the ERGs were fully developed, a program called Verification and 
Validation was implemented. Each guideline was verified to be written 
properly based on the generic Writers Guideline, and they were also 
verified to be technically correct. After the verification process, 
the ERG network was extensively exercised during rigorous scenarios on 
a plant-specific simulator. These exercises allowed the ERGs to be 
used in a realistic control room environment by trained operators. 

The exercises were carefully evaluated by mul ti=discipl ined teams con- 
sisting of operational engineers, licensed operators, human factors 
engineers, and training personnel. The validation provided an oppor- 
tunity to assess the usefulness of the ERGs during multiple-accident 
conditions. It also provided valuable data in determining if the 
guidelines had met their intent. 



67 



11.5,17 



Many volumes of material was developed to support the writing of 
plant'Specif ic EOPs. However^ the Procedures SubcomiTii ttee (now the 
Operations Subconinii ttee) recognized that the focus was mainly on the 
development and implementation of the ERGs with little thought given 
to operator training. The ERGs are evolutionary in that they have 
little resemblance to the event^based Emergency Operating Instructions 
(EOIs) of the past. The ERGs and Users Guide should be considered new 
and unique to the operator who has been trained on the EOIs. Most 
operators are not familiar with a two^column format approach to proce= 
dures. Even though the seven volumes of material developed for the 
writing and implementation of plant^specif ic EOPs contained informa- 
tion of great value to operating staffs, it was not in a form directly 
usable in a formal training program. As a result the WOG funded the 
development of several training courses. 

The first training course was developed as part of the Validation 
Program for the ERGs. The operators involved with the validation 
received a one-week course that covered the Users Guide and a basic 
overview of the ERGs. The materials from this course were made 
available to WOG members as part of the Validation Report. 

Other significant WOG training courses were authorised to focus on 
some other specific issues. The first major training course addressed 
the Pressurized Thermal Shock (PTS) issue. This training course 
covered all aspects of PTS^ including additional analysis. Two 
volumes of generic materials were developed sper^-p-^^^l ly to train 
utility plant personnel. One volume^ the Jcij:; ^. y 'i^i^^^ contained 
technical material for use by the student K a" ^ ' ^ a' section con- 
tained objectives^ text to support object vv^.:^ . ^ ^' ^^y* pertinent 
references, and self^assessment questions* Inv :,d volume^ the 
Instructor Information Volume ^ consisted of lesson plans (based on 
the text material contained in the Technical Volume )^ Information to 
adapt the materials for plant-specific use, examination bank/key^ and 
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information on how to provide PTS training on a simulator. Even if the 
utility did not have the timt or the risources to make the materials 
plant specific, the generic material could still be effectively used 
to train plant personnel in the basic principles* 

Two other extensive training courses were funded by the WOG for 
training plant personnel on ERG-related subjects. The SGTR Training 
Course and LOCA Training Course were developed to support ERG 
training. Technical information and instructor Information volumes 
were also developed for these training programs along with the addi- 
tion of a third volume, the Guideline Information Volume * It con- 
tained extensive information for training personnel on guidelines 
pertinent to the training course. The SGTR and LOCA Training Course 
development programs each authorized a ona^WGek training seminar* 
These seminars provided utility training personnel with the training 
materials developed for the programs, explanations of the materials, 
and the opportunity to question those individuals who developed the 
programs. It was envisioned that the training personnel who attended 
the seminar could return to their respective utilities and integrate 
the materials into their plant specific training programs. 

The final ERG program to be implemented was the ERG Maintenance 
Program. It evolved as a means to document and evaluate important 
feedback information concerning the ERGs, Any feedback information 
that is generic in nature and has merit to improve the ERGs is 
collected and evaluated in detail. Worthwhile information is then 
systematically returned to the WOG members for possible implementation 
into their EOPs* The WOG also retains the information which may be 
used for a future ERG revision* Sources of feedback information can 
come from any utility source that is associated with the ERGs* For 
example, the disciplines of engineering, licensing, training, and 
operations have all been suppliers of feedback on the ERGs, Also, a 
large amount of feedback material was generated when the Revision 1 
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ERGs were validated, and more feedback came when plants performed 
their own plant-specific EOP Validations. 

The WOG also supported the ERG Program by sponsoring a seminar for 
NRC-Hcensed operator examiners* The NRC seminar provided the NRC 
examiners with a better understanding of why the ERGs were written in 
their unique format and how the ERGs are expected to be used* The 
seminar also provided an opportunity for the examiners to ask ques- 
tions and clear up any misunderstandings. Simulator demonstrations 
were also run to show the effectiveness of the ERGs during major plant 
accidents* 

The ERG Development Program is an excellent example of how the WOG 
has provided utility training departments direct support in preparing 
their operations staff for EOPs implementation. In return, the 
training staffs provided important feedback to the WOG on potential 
problems with the ERGs* This mutually beneficial arrangement can only 
strengthen the usefulness of the WOG and improve overall plant safety* 



RECOMMENDATIONS TO IMPROVE THE BENEFITS OF THE WOG FOR TRAINING 



The WOG is very well organized and managed by competent and exper-- 
ienced utility members. However, problems do arise^ especially in the 
area of communications and dissemination of technical information* 
The WOG membership is international and thus encompasses a wide 
variety of utilities. Many utilities are unable to send proper 
representation to the WOG meetings and seminars* Even though all of 
the important information developed by the WOG is sent to utility 
representatives, the significance of some of the material may not be 
fully understood by the utility representative* The receipt of WOG 
documentation by the utility representative does not mean that it will 
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end up in the possession of the individual or group who can most 
effectively use it* This problem can be alleviated when the owners 
group itself specifies or recommends those utility departments or 
groups who could most benefit from the information. The utility 
representative couTd then ensure that the material got to the 
appropriate group. Greater use of the documentation will generate 
more beneficial feedback to the WOG whether it be in the form of 
questions and/or recommendations. 

Another area involves utility training department resources and 
limitations. Since the WOG generates a tremendous amount of 
technical materials it is difficult for training staffs to filter all 
the information and integrate it effectively into training courses* 
This situation can result in vital technical material not being used. 
The end result is that the utility supports the WOG and yet does not 
fully utilize the benefits of such an organiiation. 

The importance of personnel training has risen steadily since the TMI 
accident, especially in the areas of plant operations and maintenance 
Utilities have been required by the NRC to upgrade their training 
programs significantly and to invest in new training resources and 
facilities. One-time investments of tens of millions of dollars is 
not unheard of along with annual training operation budgets of 
millions of dollars. Utilities must now conform with INPO accredi- 
tation requirsments and seek standards of excellence rather than the 
minimum requirements. To this end, the WOG can contribute readily 
for its member utilities. Many training Issues are generic in 
nature. A well=developed set of training documentation for one plant 
or a generic plant can easily be modified for a specific plant. 
Highly technical or unusually difficult or unique training require^ 
ments can be handled most efficiently and cost effectively by a joint 
organization. This situation could be Improved If utility training 
departments established a special organization that is focused on 
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training within the WOG structure. It is suggested that a Training 
Advisory Group (TAG) be chartered to function within the WOG structure* 
This group would consist of utility training representatives that would 
continuously review the technical information developed by the WOG and 
make recommendations as to how the materials could be implemented into 
utility training programs. The TAG could interface directly with the 
Steering Committee and Indirectly with th^ indi /vdual subcommittees. 
Any program that may be founded by the WOG could be sviewed by the 
TAG and recommendations could be provided on the training aspects of 
the program. These recommendations could include no training neces^ 
sary, training recommended but developed by the individual utility^ or 
a generic training program developed by the WOG for use in utility 
training programs (similar to the ERG Training Programs). The TAG 
could also receive direct input from the utility training staffs on 
training issues and programs that may help other member utilities* 
The TAG could evaluate the training program and assess its feasibility 
for generic use by the WOG and then present its results to the 
Steering Committee for review. If the Steering Committee determines 
that the program requested by the TAG is worthwhile 5 then it would 
send 1t to the WOG membership for approval by vote. The TAG could 
also determine which subcommittee or group could best develop, 
implement, and monitor the program. It would provide the WOG with 
training representation and insight for all the WOG funded programs. 
The actions of TAB would ensure that utility members would receive 
Subcommittee reports with training recommendations and/or documen- 
tation, 

CONCLUSIONS 

The NSSS Owners Group provide their utility members with shared 
technical expertise, shared experience^ and shared costs. The overall 
financial savings to each member for the diverse programs funded by 
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the Owners Group is substantial. The Owners Group provides its 
members with excellent technical materials to improve plant oper- 
ationSj design^ maintenances engineering^ and training. This 
technical information should be used more effectively by individual 
utility training departments to improve training programs. The 
effectiveness and credibility of utility training could be further 
enhanced if training was spotlighted more in Owners Group. A special 
training group could advise the Owners Group from a training stand^ 
point as to how funded program materials can be better implemented by 
utility members. It could also recommend generic utility training 
materials and courses for funding by the Owners Group. The overall 
result of enhancing training quality within the nuclear industry would 
be improved plant operations and safety and greater public acceptance 
of the Nuclear Power Option. 
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NRC INFLUENCES ON NUCLEAR TRAINING 



Oohn N. Hannon 



ABSTRACT 

NRC Influences on utility training programs 
through prescriptive requlrenients and evaluation 
of Industry self-initiatives are discussed. 
NRC regulation and industry initiatives are 
complimentary and in some instances industry 
initiatives are replacing NRC requirements. 
Controls and feedback mechanisms designed to 
enhance positive NRC influences and minimize 
or eliminate negative Influences are discussed. 
Industry and NRC efforts to reach an acceptable 
mix betwien regulatory oversight and self- 
Initiatives by the industry are recognized. 
Problem areas for continued cooperation to 
enhance training and minimize conflicting 
signals to industry are discussed. These 
areas Includes 

requal if ication examination scope and content, 
depth of training and examination on emergency 
procedures; Improved learning objectives as the 
basis' for training and examination, and severe 
accident training. 
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PERSONNEL SELECTION AND EMOno_NAL STABILITY CERTIFICATION: 



ESTABLISHING A FALSE NEGATIVE ERROR RATE WHEN CLINICAL INTERVIEWS 

ARE DONE "BY EXCEPTION" 



Phnip E, Berghauseni Jr.^ Ph.D* 



ABSTRACT 

The security plans of nuclear plants generally require that 
all personnel who are to have g/iescorted access to protected 
areas or vital islands be screened for emotional instability* 
Screening typically consists of first administering the MMPI 
and then conducting a clinical interview* Interv iews-by- 
exception protocols provide for only those employees to be 
Interviewed who have some indications of psychopathol ogy in 
their MMPI results, A problem arises when the indications 
are not readily apparent^ False negatives are likely to 
occurj resulting in employees being erroneously granted 
unescorted access. The present paper describes the 
development of a predictive equation which permits accurate 
identification, via analysis of MMPI results, of those 
employees who are most in need of being interviewed. The 
predictive equation also permits knowing probable maximum 
false negative error rates when a given percentagR of 
employees is interviewed* 



Emotional stability screening is valuable in helping to ensure 
reliability on the job* It is also useful for maximizing responsiveness 
to training. The psychological factors affecting the former are likely 
to be largely the same as those affecting the latter. The person who 
is burdened by emotional problems Is not likely to benefit as much from 
training as the person who is not experiencing such a burden. 

The security plans of nuclear plants general ly require that al 1 
personnel wno are to have unescorted access to protected areas or vital 
islands be screened for emotional instability. In virtually all 
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instances, the screening involves the administration of one or more 
psychological tests, usuany including the Minnesota Multiphasic 
Personality Inventory (MMPI)=~a 566-item true^false inventory designed 
to detect serious psychopathol ogy. At some plants, all employees 
receive a clinical interview, typically conducted by a psychologist, 
after they have taken the MMPI and fesults have been reviewed. At other 
plants, only those employees with "dirty" MMPI's (ones with some 
indications of possible psychopathol ogy) are interviewed. This latter 
protocol 1s referred to as "interviews by exception." It is the 
protocol specified i'n Revision 8 of NUMARC's Industry Guidelines 
for Nuclear Power Plant Access Authorization Programs.^ 

Interviewing all employees, regardless of MMPI results, is the most 
desirable protocol, as th,e author has argued elsewhere.2 it is the most 
consistent with learned opinion In the field of psychology and with the 
known limitations of psychological tests. Ethical problems may exist 
when significant decisions are made solely on the basis of the reriults 
from a single test. (There 1s some disagreement on this issue among 
professionals in the field, and there is evidence that what constitutes 
an acceptable standard of care is changing.) Employee objections to the 
screening generally are fewer when everyone is interviewed: No one is 
singled out as "marginal," the results of an admittedly imperfect test 
are de-emphasized, and there is an opportunity to get some feedback 
regarding the test results. 

Perhaps most important of al 1 is the empirical evidence derived 
from actual experience with nuclear plants which demonstrates clearly 
the potential for error that exists whin interviews are omitted. 
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Behaviordyne Psychological Corpo*"ations which provides emotional 
instability screening services to the nuclear Industry^ studied the 
results obtained from approximately 19,000 employees screened over 
almost a 10-year period* All employees who were screened were 
interviewed-'-not only those with dirty MMPrs, It found that 38% of all 
employees whom it failed to certify for unescorted access (1*e,p 38% of 
all "noncert if ications") had MMPI results devoid of clear indications of 
the presence of psychopathol ogy. In testing Jargon^ these were "false 
negatives." Had these persons been screened by an inter v iews-by^ 
exception program, many of them (unwisely) probably would have been 
granted unescorted access to nuclear plants, (Ironically, although the 
program to screen for emotional instability was designed to protect 
property and the safety of co-workers and the surrounding community, 
interviewing by exception would appear to emphasize protection of the 
rights of an individual employee to the relative exclusion of the other 
protections that it was created to provide.) 

If an inter views^by^except ion protocol is implemented, it becomes 
extremely Important to identify accurately, via psychological testing, 
whom to interview: Anyone who 1s not interviewed will be given 
unescorted access without any sort of further review, Behaviordyne 
decided to see whether it could develop a predictive equation^ which, 
when applied to MMPI data, could assure that all persons who should be 
interviewed would be, that persons with significant psy chopatho 1 ogy 
would not go undetected by the MMPI, and that false negatives would be 
reduced to an acceptably low (and known) level. 

Fortunately, Behaviordyne had a very desirable pool of data with 
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which to work. Not only was it large (19,000 cases), it was diverse. 
It included data from employees in a variety of professional /managerial , 
skilled, and unskilled Jobs working at nuclear plant'; across the 
country. It included a wide age range (teenagers throii- ■ - in their 
70's) and spanned almost a decade. Most importantly, Lhere was 
interview data on every person who had completed screening, including 
those with clean MMPI results— thus permitting study of potential false 
negatives. 

The protocol by which all of these employees were screened was as 
follows: Each employee was administered the MMPI. The MMPI results 
were read and interpreted by a 1 icensed psychologist. If the results 
were clean, the employee was granted an "interim certification," which 
permitted unescorted access for up to 90 days. If the results were 
dirty, the employee was given an "interim noncertif ication," which did 
not permit unescorted access. Regardless of test outcome, the employee 
was interviewed, typically within several days to several weeks of the 
test administration, by a psychologist who sought to ascertain whether 
the employee met N.R.C.-specif ied criteria for emotional instability or 
behavioral unreliability: argumentative hostility toward authority, 
irresponsibility, defensive incompetence, adverse reaction to stress, 
and emotional and personal inadaptabi 1 ity.^ If he or she appeared to 
meet one or more of these criteria, a noncertif ication would result—but 
only If there was concurrence by another psycho! ogist— and unescorted 
access would be denied. If he or she did not appear to meet any of 
these criteria (or if there was no concurrence from a second 
psychologist), certification resulted; The employee was granted 
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unescorticl access. (Unlike the Interim certification, this 
certification subsequent to the interview carried no expiration date.) 
It was possible (indeedp probable), as the data below indicate, for an 
employee who had received an interim noncertif icatlon to receive a 
certification upon being interviewed, (A person can have psychological 
problems that pose no risk to a nuclear plant.) Likewise, it was 
possible (though relatively rare) for an employee who had received an 
interim certification to lose this certif icationp as a result of 
interview findings, and receive a noncertif icatlon* 

A summary of the results of the screenings is as follows: 

----1% of all employees received noncertif ications 

^=78% of all employees received certifications 

^^21% failed to complete the screening process (were laid off, 
terminated for cause, eliminated themselves from further 
consideration because they had received an Interim 
noncertif icatlon, became 111, etc) 

---12% of those who failed to complete the screening process had 
received interim noncertif ications 

^•-12% of all employees received interim ^'^ncert if Ications 

— 88% of all employees received interim certifications 

--38% of all employees receiving noncertif ications had clean MMPrs 

(i.e., they had received interim certifications) 

--62% of all employees receiving noncertif ications had dirty MMPTs 
(1,e,, they had received interim noncertif ications) 
---6% of all employees who had received interim noncertif ications 
and completed screening received noncertif ications 
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—0.6% of a1 I employees who had received interim certifications and 
completed screening received noncertif ications 

—94% of all employees who had received interim noncertif ications 
and completed screening were certified 

—99% of all employees who had received interim certifications and 
completed screening were certified 

Included in the present data pool used to create a predictive 
equation were the MMPI records of all 233 empl oyees who had recei ved 
noncertiiMcations. An approximately equal number of MMPI records of 
employees who had received certifications were selected essential ly 
randomly, taking care to see that all years and plant sites were 
represented in proportion to their representation in the total data 
pool. These latter MMPI records included those that had elicited 
interim noncertif ications as well as those that had elicited interim 
certifications. Excluded were the records of employees who had not 
completed the screening process. 

The criterion variable was the clinicians' decisions regarding 
(final) certification or noncertif ication (i.e., the decision which 
followed the Interview). Although the accuracy of clinician interview 
Judgments, and their appropriateness for use as a criterion variable, 
has come under much attack over the years, there is considerable 
evidence in defense of their accuracy when the decisions made are global 
ones (as they are in making certification/noncertlf ication decisions).^ 
Several studies have demonstrated empirical ly that Interviews in 
conjunction with psychological testing contribute to the overall 
accuracy of the decisions which result.^'^ In addition, in the present 
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studyp all decisions leading to noncertlf Ications were confirmed 
(although not blindly) by a second psychologist. Further reason to 
believe in the accuracy of clinician judgments is that^ to the author*s 
knowledge, employees who have received certifications have never baen 
linked to sabotage in any nuclear plants. The most serious 
transgressions have been limited to illicit drug sales and/or use, 
(From a methodological point of view, the best validation of clinician 
decisions would come from granting unescorted access to all employees--- 
both those who had received certifications and those who had received 
noncertif Ications-'-and waiting to see what problems arose. Obviously 
the potential costs of such an approach, not to speak of the ethics, are 
totally unacceptablep ) 

The MmPI responses of those employees who had received 
certifications after being interviewed were compared to the MMPI 
responses of those employees who had received noncertif ications. The 
data were analyzed using multiple discriminant function analysiSj a 
technique generally more powerful than multiple regression analysis. On 
the first pasSj only existing MMPI scales (a scale is a group of items 
which measure a common construct) and indices (an index usually is a 
score that results from a combination of scales) were used as predictor 
variables. In order of predictive strength, the nine with the greatest 
loadings were: Social Maturity (index), Addiction Band (Index), Meehl^ 
Dahlstrom Rules (index), ACG Codes (Index), Psychotic Point Count 
(index). Hysteria Score (index). Responsibility Averagn (Index), MaK (K^ 
corrected Minnesota standard score on scale 9, the Hypomania Scale)i and 
Alcohol Band (index). 
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It should be noteri that only one of these variables, MaK, appears 
on the original 10-cl Ira ical -scale version of the MMPI. (Scales 4 and 6, 
Psychopathic Deviate and Paranoia, both among the 10 original scales, 
did manifest some predictive power on their own but not in conjunction 
with the other 9 predictor variables.) Thus, MMPI scoring systems which 
yield only the 10 pyiginal clinical scales are Inadequate for 
ascertaining accurately which employees should be interviewed. 

Using these 9 vayiables in a predictive aquation yielded an 
accuracy rate of 78%s i.e., 78% of all employees were correctly 
classified by the equatTon. A second pass through the data was made to 
see whether this accuracy rate could be improved upon by using 
individual MMPI item T-esponsts, as well as scales and indices, as 
predictor variables. Forty-four variables with significant loadings 
were identified, most of them being Individual items. Using these 
variables in a predictive equation, a accuracy rate of 87.5% was 
obtained. Cross validation using a Jack-knife technique resul ted in 
only minimal shrinkage: The accuracy rate obtained was 85.3%. Present 
indications are that 30 percent of employees will need to be Interviewed 
in an Interv iews-by-ex ceptlon program which uses the cut-offs that 
yielded this accuracy ra^e. 

The predictive equ ation will have differing accuracy rates and 
types of errors (i.e., tKe relative proportions of false negatives and 
false positives will chsnge) when different cut-offs are used and when 
different percentages of employees are interviewed. 

CI lent companies, tooth .ut1 1 ities and vendors, w1 1 1 be abl e make 
Informed cost/benefit analyses by using this predictive equation. They 
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will be abl e to know the maximum number of Ise negatives likely to 
result 1f a predetermined percentage of eni^ 1 oyees is interviewed. 
Conversely^ 1f they decide upon an acceptable maximum number of false 
negati viSs they will be able to know the percentage of emp 1 oyees who 
will have to be interviewed to assure that this maximum is not exceeded. 
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aP&LICATION OF SUHAN FACTORS DESIGN REVIEW 



PRINCIPLES mm TO TI^INING 

D ^ R, Duquette 
D* Gaddy 
T, Martin 

ABSTRACT 

D^fcailed controi room design reviews have 
been c^&ducted at nuaXaar power plants to evaluate 
huitiln f^c^tors conaidttrations in the control 
cooni. ^Euman factors is a diiGipline dedicated to 
Imptovirmg systeni deitgn from the operations 
peflpe^t^ive » The hum^n factors principles 
Qonaider— during ths design reviews are also 
^ppUc^l»le to trainir^g* In addition, much of the 
^afcl g^n^erated during the design reviews can be 
lis^^ by training pec«onnel in the development of 
lei^on ^lans and in slassrooni or simulator 
preient^ tions , 



^ major hum^rw factors initiative in response to the Three 
Mile Ealan^ incidfe^t has bian tlie performanGe of detailed control 
room aesign mi&\r^m (DCmRsj im. nucleac power plants* The U^S* 
NUcl^»r ReguJatg^y^ Commissian ir squired that DCRDRs be conducted 
and pabHshad "Gt^i felines for Control Room Design Reviews" (NUREG- 
0700).^ The pUrpPa m of the DCROKs was to review and evaluate 
contfc^l tpom desig: ^ from a humaxi factors point of view. The 
objective wai to irmprove the c^^trol room from the operator's 
pecapsctives Mo^t nuclear powsr^ planti^ to our knowladga, have 
completed DcRDRs a ^ this tima. Many of the plants are in thm 
ptDcess of ifflplertier^iting controL room modifications based on ttie 
results of the DCft33H* Also^ earrently^ the focus of human factors 
attent^ion has eKpt=mded f rom cora. trol room design to encompass 
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control room procedures, and from the control room to other work 
areas in the plant. 

Human factors is a discipline dedicated to improving system 
design from the user's perspective. During the design process, 
decisions are sometimes made without the user in mind^ or without 
giving due consideration to the user's eKperience, work 
conditions^ and task demands. As an eKample, controls may be 
arranged without consideration of their use, A classic example 
involves an aircraft cockpit design in which the seat ejection and 
seat adjustment levers were inadvertently swapped. This design 
change obviously inter f erred with the pilots expectations for the 
lever location based on their previous eKperlence, Even if the 
pilots could have b.een retrained to use the correct lever for its 
Intended function, behavior under stressful work conditions often 
reverts to previously learned responses that have been practiced 
eKtensively in one's past experience. Consideration of the users 
expectations during design, or system modification, can usually 
eliminate this type of poor design decision. 

Other types of human factors problems result when designers 
have not considered human capabilities and limitations. Humans 
have remarkable mental capabilities, but we also, have our 
limitations. Operators can remember impressive amounts of 
technical knowledge, but short=term memory should not be 
overloaded. For example, requirements for an operator to make 
complex mathematical calculations in his or her head is usually 
unneuessary and will often result in errors because of the 
difficulty of performing complex math mentally. A human factors 
recommendation might be to provide a table or other written aid to 
assist the operator In performing calculations. Then, the 
operator's memory can be focused on the plant situation rather 
than on mental math. 

Humans have physical capabilities and limitations as well. 
At some plants, we have found reguirements for control 
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manipulations that were uncomf or table # if not impossible. For 
eKample^ small pushbuttons that must be depressed for a long 
period of time. In this ease, some operators' fingers were too 
large to depress the button. The operators that could push the 
button, reported that their fingers got uncomfortably sore holding 
the button down foi: the peric^ of time required. Once again, the 
user was not considered sufficiently during selection of the 
controls , 

The purpose of this paper is to review the human factors 
principles considered and data generated during DCRDRs, and 
discuss the potential applications of the principles and data to 
training. Seven major DCBDR activities will be discussedi (1) 
system function review and task analysis, (2) operating experience 
review, (3) control room inventory, (4) control room survey, 

(5) human engineering discrepancy (HED) documentation, 

(6) evaluation of proposed design changes, and (7) human factors 
plan for future design modifications, 

SYSTEM FUNCTION REVIEW AND TASK ANALYSIS 

During the system function review and task analysis portion 
of the DCHDH, the functions of plant systems were reviewed, 
operator tasks necessary to support system functions were 
documented^ and information and control requirements needed by 
operations personnel to perform the tasks were defined. 
Documentation generated during the system function review and task 
analysis can be useful to training instructors as a source for 
"big picture" information about system functions and 
interrelationships, and the operator's roles in plant operation. 
The delineation of operator tasks could be used to identify 
systems operated during emergency tasks, and critical tasks 
performed during plant emergencies. Both these applications may 
be useful in developing or sequencing lesson plans to support 
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emergency operating procedures training. Further, the task 
analysis can form the foundation for, or serve as a cross check 
of, the job analysis performed for training purposes. 

OPIRATING BKPERIENCE REVIEW 

The operating experience review (OER) portion of the DCRDR 
involved gathering historical information on control room 
operations with the focus on problems that Involved human factors 
considerations (primarily at the operator-machine Interface) . 
Sources of Information such as licensee event reports were 
reviewed, in addition, operations personnel completed 
questionnaires and were interviewed to provide a source of 
historical Information the operators could contribute based on 
their experiences. The OER generated examples and case studies 
that can be used in training. For example, at one plant, an event 
report stated that operators were unaware of the reason for a main 
feedwater pump trip which subsequently resulted in a reactor 
trip. DCmm recommendations Included annuclator and procedural 
changes. Until the problem is remedied, this type of example 
would be a good case study to be covered in training. 

Further, lers or other plant Incident reports may have 
attributed the cause of the problem to operator errors as a result 
of training or performance. In fact, human factors design 
problems might have led to the error, when the LBR or other 
report was written, the author may not have been familiar with 
human factors design problems that could have contributed to the 
incident. The best trained, most competent operators will still 
commit errors if the panels, instruments, or controls are very 
poorly designed for human operation. 
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CONTROL ROOM INVENTORY 



The control room inventory was developed to provide DCRDR 
team members with a database of control room instrumentation and 
controls (IsC) against which to compare information and control 
reguirements generated during the task analysis (this process was 
referred to as "verification") . The inventory can provide a 
unique source of information on characteristics of plant I^kC. For 
example^ trainees may have questions about the range ^ scale # or 
multiplier for a meters or about the functional detentes or range 
on a controller. This information may or may not be obvious when 
observing the panels. In any case* the information may not be 
readily available at a training center. 



The control room survey portion of the DCHDH involved the use 
of checklists and other data collection forms to evaluate the 
control room against human factors guidelines fori workspace * 
communications ^ annunciators , controls # displays, labels^ 
computers, and panel layout* 

Much of the information collected during the survey was 
independent of any particular operating sequence or scenario. For 
example, if one pair of indicator lights was found to have the red 
and green lens covers swapped incorrectly, an HED would be 
recorded regardless of the operational conteKt in which the 
operators use the lights. At a later stage of the KRDR^ 
walkthroughs were conducted in the simulator, control room, or at 
a mock-up. These walkthroughs provided an Opportunity for DCRDR 
team members to record human factors problems in a dynamic, 
operational context (this was referred to as "validation") , Some 
control room survey items were further clarified or found during 
the walkthroughs. 



CONTROL ROOM SURVEY 
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ra^e results o« the control room survay include documentation 

of obv^ouaand non— obvioua design principles used during initial 
board design, or to be used during control room upgrades. 
Principles such as arrangements of controls and displays based on 
function, ecitldalLty, sequence of usa, or frequency of use will 
be of interest to training instructors. These principles can be 
usea provide an organizational scheme for the instructor to use 

In presenting train ±ng material. For example, if a panel 
arrangement of IsC laid out to be consistent with the system 

Eumtl^x. (using a m ^mic or other location cues), an Instructor may 
want to. discuss the logic behind the control panel layout for the 
systefn ^hen he or st^e is teaching the system. 

Ah adaitionai, related benefit of using the human factors 
Peinclp-Le, is that provides the operators with a mnenomic. 

I.e., memory aid, tc^ use in remembering the logic of anel 
IVouts. With the assign principle as a memory aid, the operator 
may fina it easier to locate isC or remember the function, 
ccltica^ityor sequa nee of the operation. 

Ottaer human Sac tors design conventions for location, color, 
size. 0^ shape coain^ niay not be readily apparent to the new 

HtJ^tAN EMGINEERING DISCREPANCIES 

Deviations frofi human factors principles that were found 
during ttie DCRDR activities were documented on human engineering 
discrep^r^ay (HED) £a^^^. The HEDs that will result in changes to 
m control room incL«de a justification for the modification that 
can u«ed to Juatir^ to operations trainees or requallfli=catlon 
crews th» basis for ca^anges to the control room. HEDs for which 
no ammr^ changt was ^ecom^r.ended also have a justification which 
the enper ienoed opeua^ors may be especially interested in since 
they theni.^elves may h«ve suggested the change. At one plant, for 
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eKamplef (pPerators recomrnemded major changes-, to the annunciator 
system, ^hmthB utility L nstrufnentatlon #1^-^ control personnel 
found th^t at that point irm. tlmey several ye ^ ^rs agO/ no vendor 
could supply the n^ceesary liardware* this m^m explained to the 
operations pBmnn.ml, Port tinately^ two inn^^^ative individuals at 
this plant configuired a soifcputeriied aid short«tarm solution 

to the ptot^lifii rtie utility manigement supp'^s^rted the aid 
enthusiastically. If the i-evalopers of the «^omputer aid had not 
bean given ^egdbaalc about tiie unavailability ©f vendor hardware, 
they might have attributed the lack of changes in the annunciator 
system to a lack of support by management* T&ian, they might not 
have been mo^tivatea to crea -fca the solution tfciat they did. 

Finally, "fi^^a" to so^ia of the HEDs in^^olve training. If a 
disarepaney involves a IqW probability of erteror and low safety 
consequance% , no design change may be warran^^ad. In these cases, 
and selacta<3 other Instanceo , training may h^m invoked as a 
solution joor design. Tfaa rationale Is threat alerting operators 
to design flaws wlXl increa»a their attentionrn to the problem so 
they can ce^^njansate for the human factors pf^^blem* One problem 
with this the effect of »trsis that was rn^ntioned early in this 
paper. In StMiifiaX situations ^ the operatoj^a may be distracted 
by plant ©Vtnts an^ forget to coinpensate fot the poor human 
factors d^iign. 

EVALUATION Or PROPOSED CHA^3^ES W IMPACT ON OPEHATOR PERFORMANCE 

Evaluation of proposed c3asign changes rti^ be accomplished at 
the simulator or at a photographic (or CAP dr awing) mock=up. 
Operators In trainirig may asked to provide comments regarding 

the proposed changes* They should be encout'a^^ad to give feedback 
on the dynamic operational implications of fch*^ change. An 
InstruotQir m^ywant to obger ^a walkthroughs the simulator or 

mock-up to assess tlia poaitl -u-a or negative fcr,^nsf'ir that may occur 
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after the change Is made. Positive transfer refers to improve- 
menta in operator performance in a new situation baaed on a 
previously learned aituation. For example, assume an operator 
worked at a nuclear plant that had a good panel layout from a 
human factors perapective, and then went to work for another plant 
at Which the same panel was laid out poorly, if the plant at 
which the operator now works changes the poorly-designed panel to 
reaaemble the good layout at the former plant, the operator would 
be expected to perform well on the panel. This is an example of 
poaltive transfer. Negative transfer refers to the opposite 
situation in which the old way of performing a task interferes 
with the new. For example, if an operator has worked at one plant 
for years and has learned to work with a poor panel layout, if the 
panel layout is improved, one might expect at least an initial 
performance decrement. Thia operator would be exhibiting negative 
transfer because the old way of interacting with the panel might 
interfere with the new way. DCRDR team members were tasked with 
the responsibility of evaluating the potential for negative 
transfer of corrections made to HBDs in the control room. For 
some HEDs. a trade-off was made between the advantage of any given 
improvement against the disadvantage of negative transfer. 
Fortunately, for most corrections, HEDs improvements will be 
clearly advantageous. For example, clarifying nameplate label 
nomenclature or reducing high control room noise level, would be 
expected to improve performance in most cases, or at least not 
worsen It. 



ONGOING HUMAN FACTORS PLAN 

A final product of the DCRDR is a plan for an ongoing human 
factors program so that design changes made in the future take 
into consideration the human factors gu.«elines applied as the 
result of, or before, the DCRDR. The plan includes plant-specific 
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human factors principles to ba used by design engineers (and human 
factors engineers for major design changes) . Trainees should be 
encouraged to provide input regarding positive and negative design 
eKamples to be incorporated in the plan. 



SUMMARY 



In summary * this paper has provided an overview of the 
information generated during DCHDRs that can be useful to training 
instructors and instructional materials development personnel. 
Information related to the human factors review can be of 
instructional value in the content portion of lesson plans* for 
example^ to provide the '^big picture" of operator functions and 
tasks. In addition, the human factors guidelines used during the 
DCRDR* such as arrangement of laC based on criticality or fre- 
quency of use, can provide organizational schemes for presentation 
of information in lesson plans. Finally, Instruators with know- 
ledge of the DCRDR can answer trainee questions such as "what is 
the rationale behind control room panel I&C arrangements?", "Why 
did the control room change, that I recommended, not ^et 
implemented?", or "Why did the control room change?'' 
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THE EXPftN DINS ROLE OF OPERATIONS, TRAINING 



Mar^ Bernardi 
EXPRESSWORKS INTERNATIONAL* INC^ . 



ABSTRACT 



The "Ttunibir of Hcmnmmd nucliar power plants pr oducing commercial 

rlectrici %y isclimbingp Attitudes and skills succ e^ssfully applied to 

developi)"*q the industry are lesi successful ly appli md to managing plants 

in a st^a. dy-itate mode operations^ Fine'-tuning "the complex 

rel at i Qna^ ^ Ipi between pE-opl^ and technQlGgy is a ch ^llenge that requirei 
a fresh p ^ripictive* 

An i TicriJiiing number" of industry guidelines irfer^icate that effective 
teamworkj conimimi cat i on, leadership! and diagnQstic skills are aspects of 
human per ^ormance requir i ng attention* A totaJ sys "tern view is suggestBd 
as a ft aRtewprk to integrate these training need*^ wi the e><iEting base 

of technical instructiDn, The tptal system perspec tive balances the 
needs of people and the needs of the physical plant against the unifying 
objective^ of prof i tab l©i ^teady^state operations. ^his perspective 
favors a leadefship role tar the training function. 
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INTRODUCTION 

Th- m nuclear power industr-v-y has put together sv#t«of InCrfredible 
mqm^um^B and iQphisticatiQn . Mot only ar^ nuclear ggneratinq ^^Uiioni 
tEChnic^^Uy and michanica 1 1 y ^sompl&x^ they are individualistic ^^nd 
fickl©. Each plant contains elements of diversity and unpred icfe^ ability 
that t><^eed fchi automated system's capability to sUstaiMnd q^r— rect 
itself. Human inter vent ion required when the llfnltiof the#%^ 

Eapab i 1 ^S. t i es are reached* 

We need to understand and manage the compl&H nmtmrk of 
reUtiprteships bitwien people a^ rid the automatBd techtiol ogy they h.^sve 
built. The relationship betwe^en people and machintB iti nuclear g powir 
plants i s one of tension and r rigorous performance pressure, Th©^ e 
challeng*^ for the industry im «o reconcile these confUtts and l^earoto 
work eff^^ctively in an envlronmient that is saturated ^ ith tech n^tf logy. 

CHWNSIN3 MODES OF QPSRATIPN REQUIRE NEW PERSpKTIVEB 
As i^i-^e shift our focus frp/r^ start--up activities to lustainir^^^ 
Dpifat ipr— ts, certain conditions and problems emerge, Thi tichnac4Al 
Ikills* ^ttitudGi, and models t - hat we apply successf uUy to bu i Id i npn 
industry are no t automat ica 1 ly transferable to the chaLlinge o-f 
itiblligl_nQ an induitryp 



The Oynarnlc^s af 

As ontrihutori in the star-r-t-up 
tichnicsl ly resourciful i innova^^ivei 



Industry Start-up 

phases people ar# incouragtti ^ to be 
and driven toi^ard the iccomp; li# 
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ment - of Epecific mileBtoneB. During start-^upf thm challenges are 
visib -le. Our efforts are continually rainforced by the material evidence 
Of pr- oyress. Individuals closely identify their work efforts with 
tangit ble results and are motivated by the prospect of finishing things. 



Shifting To Steady=State Operations 

The elusive challenge of generating a favorable return on assets 
©mpha^ siies different skills? attitudes and operating principles* The 
©mpha* mis shifts from putting pieces in place to getting the pieces to 
wori^ ^ together^ smoDthly and profitably.- 



= In the sustaining mode of operations? retention of staff and human 
motiv^^tion are compelling issues. The quality of the work environment 
gaini importancE- as the intensity of start-up fades. Constant pressure 
to at^^ain a favorable return on assets often reinforces reactive and 
authors i tar i an management styles* These styles alienate the more 
i ndfip^Bndent i creative thinkers, and bring a "we/they" attitude into the 
work ^»nv i r Dnment * These conditions increase our sense of isolation and 
fBduc^» the level of satisf action we derive from doing our jobs* 

' "*Hours of boredom interrupted by moments of sheer panic is a 
phrases used by airline pilots to describe their Job* This line gets a 
nod o^tf" ligreement from eHperienced control room operators* The operators 
jnd pLfiant staff who are responsible for sustaining the plant have o 
differ" — ^ent relationship with tht^ plant's systems than those who built 
th*pni> There are frustrations with design decisions tf'iat cauc.e unforeseen 
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dif f icul tit?s and inconvenlinces , Evintf» occix^r simul taneously * in 
patterns tha« were not predicttd. Thergi is t he subtle recDLjniUon f.hat 
technology di^minates the envirDnfrie--nt and that the individual is 
subordinate* Th# day-to-day visjb jlity of en^^ resuits and personal 
identification with a finiihed product iire diwninished. 



Cl^anging Modes Of Operi^ ion RaisM Basic QuBSiUons 
NON thata w© are heref facing thirty to f^^rty years of Bteady- 
state op^rati ons, how do wi iPiet t^^ie needs of both the machines and the 
people that /it^ake up our plant system? How do uie ensure the highest 
probability o achieving the ob jecte ive of pror^itable steady-state 
operations? What are the mt heapful attitia^des to balance technology 
and people in an ef fectivei long-tsrffl relat iof>^ship? 

THE FRiVAILII^G ALTERNATI^ VESi 
THE MECHANICAL WORLD VIEM VSRiUS THE HUMANISTIC WORLD VIEW 

Our attitudes shape the way ir*« which we t»ink and determine the 
alternatives tehat are availiblt to ui. Our at-^itudesf considered as a 
whole, represent our view of the wo rid* When v^e consider the nuclear 
facility only from a mechanical vi©^^, our appr^oach to problems is 
mechanical in nature and wiari> oye^ timej on r — iuman limitations. When we 
view the plant strictly as a human Dperat ion , involving some technology, 
we overlook c& ncerns that are viei b^i? from the mechanical systems 
viewpoint. Ho '^'^N do we baldfiCi thsse fitcitudes ^^about people and machines^ 
when each repr--esents a vi©fegQf the jorld that ^=Dnsiders the other to be a 
liability? 
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The Mechanical World View 

Th© mechanical world view Bmphaslies Ihi objective rt^-alityof t^e 
physical plant. The plant* as an i ntr i c#te arrangBment of compQnents and 
submyst©mss demands Epecific treatment. PrfcisiDn im eSstw^tial for the 
plant to operate in a manner that appro^lmatii its design ^specifica^ 
tlons. The terms and CDnditions of this triitment art dese^ribed in sn 
t -growing* ever^changing array of S.O.P.s, industry reg&JlationSs and 
diagnostic procedures. These documents outline the specific roles ai^d 
responsibi 1 i t ies of all licensed operators and support sta^^f. The 
implication is, that if people do exactly what is called f«=rj when it is 
called for* the plant will operate itself. 

The viability of the mechanical world view restE on s^^veril primary 
assump t ions i 

o People will continue to assimi late and recall mof^s prDCedur»^ 
and specifications accurately. 

o Procedures are current and reflect the actual con^^ition of tfrfe^ 
p 1 ant , 

0 Properly trained* people will execute the procedur^e corrective 
every time. 

o The impact of human personality on performance is neutralized 
by procedurally scripting roles. 



A basic problerii with a purely mechanical view is that it discounts 
human limitations, particularly those that surface over the long t&rm, 
and assumes that people will respond consistently to change. 

Another blind spot in the mechanical view is that it does not allow 
for the e><istence of personality. Quirks, Minks? and unforeseen 
interactions between people and the plant are analyzed in isolation and 
addressed independently as incidents or events. From the .mechanical 
perspective, we will consider the individual to be a liabilityi best 
dealt with by iEolating and limiting human intervention. 

When we were building the nuclear power industry ? the mechanical 
view helped keep us focused and on track. From the mechanical view, we 
can stop a system* look at it in still-^life, and analyse it. Once a 
system that includes people is up and running, it is no linger possible 
to freene and understand it by examining its parts in isolation. In the 
sustaining phase of plant operations we need to learn how things work 
while running at one-hundred percent power. 

The Humanistic World View 

The humanistic world view holds that the characteristics of the 
human organism are the most effective model for arranging work and 
understanding relationships. Machines, once created, are only reliable 
when actively controlled by responsible peoplo«, supported by organisa^ 
tions that reflect human values. 
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The humanistic viei^» assum©-^ .tn absuiutf? dependf^'nce off the? \3ub joc t i w'lv 
qualities and rer-ources of p^Qplff, Wh^n the? plant perturbates.. and fai]^. 
to correnit itself* when the reactor's unique persnnAHty emergec 
unoxp&ctedl y , people are required to diflgnose and intervene. The human 
element is called on to discover the best way to interrupt an undfic i r af> 1 e 
string of events that may lead to equipment damage, down-^times or a 
situation that j r^apfird i ^es putalic saf ety. 

Much rjf human factor!! engineering is based on identifying those 
i Viter sec 1 1 ons of people and machines that cause people to become 
i nef f i c ierrt s uncomf or tab 1 e ? or unreliablB due to the cnnf i gur at a on of tho 
plant and its contrDls. The tendency^ wh»sn thini^ng from tl^e humanistic 
world view^ is to find Nays to accommDdate the machine to the needs of 
pC/Op i e ^ 

A probjf^m i ntierent in the tujmanistic view is the strong evidence 
that multiple, mf^chanical accommodations may degrade the performance of 
technolog/ in une>^pectedf possibly Icgarithmic w^ve. Given the size* 
complexity* and potential impant of nuclear power plants? it. is unwise to 
look only for wciys to adapt the technolDQ/ ^o people or to the implicit 
values of the? humanistic world v^ew, 

Man versus Machinal 

Einstein pr^dictea ^^his difficuHy in the re 1 .i t i nn^^l, i n bcj^tween 
people and technology. He foresai^i that traditional spprQaohes tn 
□ rgsn? nation and problem Eolving would not hold up to th^^- lona-tfirff. t^s^^ 
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of mfflnat ing CachnoloGy and its Impact on our Uve-s. As Einatein and 
otherB Burmi-jeri, we nsHd now ways of thinking jibout oufSBlvea in rolafcioii 
to th» technological environment that we have created. A new parspectivo 
that balancea the machanical and human aapects of our lives is called 
for. 

While concentrating on th» immediata tasks of start-up in the 
nuclear powep industry, we inserted oursolves into a problem. We built 
plants that require people and technology to operate togc-thBr, in ways we 
did not 0 iginally anticipate. We are looking for answers from world 
views that give us an incoff,plete picture of tne situation. We are now 
chalienged to redefine our relationships with other people and machinBB 
while we are perched on the edge of change. The reality of the moment is 
upon us, and whatever we elect to do or not do, will have far reaching 
consequences . 

THE TOTAL SYSTEM VIEW 
Within the framework of the total system, everything that happens 
within the system has an effect on the individual i and conversely, 
everything the individual does has an effect on the total system. The 
total syBtem view is the collective identity of all the parts and 
subsystems of an operation. This collective identity encompasses and 
supersedBS the identity cf the individual components and establishes the 
basis for personal accountability and teamwork. 

What is suggested here, is an attitude about relationship that 
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tranaceirds the pQlarib/ of the mechanical jiicJ hn.r^^ruLiUc viovm. it is a 
mental frafnt-wjork that allows us to develop moiB comprehc?ns i vi5 , rt:spon'- 
sivB relationships with technology dnd with other pf?nplc?. Through the 
procw%% of applying the total system view to our t-^ork. l£?arn to 
balance our human needs with the mechanical needr, of the pl.^nt. 

The dBvelopment of a total system view cannot be instructed. It is 
transferred by modeling behavior that is correspondent with the sttitude 
of accountability. The total system view is sponsored rather than 
taught. Sponsorship is accomplished by pro^adrng personal eyamples for 
others to observe and omulatej at tfie learner's discretion and pace. 

The exemplary model emerges from the individual, and the 
organiaat Ions that studies themselves in relation to the system around 
them. This includes noticing how we feel in different situations; 
watching how we and others react to pressure^ boredom, frustrattDnl 
identifying behavior that is triggered by human interaction and 
interaction with the plant! monitoring our communication patterns and 
tendencies; watching how we respond to conflict? sarcasm, and criticism* 

When we become watc:hful of our daily activitieSs we see patterns 
that support or add difficulty in achieving our goals. We notice the 
source of conflicts* When we are capable looking objectively at what we 
are doingj we can then ask, "What or who are we doing it for?" This 
quest ion leads us to compare our personal needs with the needs of tne 
total system. From this comparison we will identify ways to bring our 
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Dbjpct;jv&? in J in© igith the prevailing objectives of the plant, Thf 
aUgnment of needs and Dbjectiv&s with thosp of the total systems iith^ 
cornerstone to a deeper sense of contribution and job satisfaction. 

When we see ourselves as contributors, as belonging to sompthing 
worthwhil© and impDrtrtnti our behavior and rommun] ca 1 1 on begin to reflect 
this perspective. We become a moriel for others to Cibserve relative to 
their experience. Dur awareness and objective pj^pression of how things 
are, becomes a catalyst for change and improvement in others. 



Meeting Industry Neede 

There are an inc reccing number of industry guidelines that 
indicate a need to train licensed operators in teamwor!', leadership^ 
ccmmunicstion. and diagnostic skills. Hany training supervisors accipt 
these guidelines as just another curricylum requirement, another 
commitment, to be satisfied. Some training professionals believe they 
have programs in place, but are getting signals that thrir efforts are 
insufficient. To others , these guidelines represent an opportunity to 
evp^riment wjth concepts and attitudes that will shape th(?ir 
orgr. -ation^s apprDach to optimising total plant perfarmance in the 
sustaining phase of operations. From the total system viei^, we see the 
fulfillment of these guideiir^es atj an opportunity to expand the rolP and 
methods of training. 

Several tjtilities are integrating the principles of teamworM. 
leadership, anet communication with their existing approac^. to training 
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and CQurss design. What they are digcovsring is ,3 prDcess that 
emphasiaem certlin prsmisssi 1, VThe total system view allows^^di 
individual t'*;3iei how their ral& ^arnpacts others and hnw othsPBHa/?? ^ n 
uffect on then 5. AwareneiiS of nterpersonal strengths rind t^^Diknos'S- .as 
helps peopl© rid Just to new team a^ss i gnments mDrf? quichlyl 3. Ihe 
attitude of accountability sttmul^ates tha acceptance of personal 
respons i 13 i I i ty for learning and if^siproving team per f Drmanc e I Trtir^ ir»g 
in list^ninQ and CDmmuni cat ion %k^^ll% rievelops empathy and considB'"3t ion 
for the needs and objectives of dt£:hersl 5, InteractlvBj partiinpane 
centered training tschniquem imp rg^vg retention and critical thinlang 
skills^ 6, Etprienced operators and supervisors ha^e a ''coaching ro le" 
in the reinforcement of training ^and skill development I 7, All 
operations andsupport staff requL_ire exposure to training that imphsi ises 
consistent appfDaches to tBaffwork** co'iimunicat ion^ and cJ iagnostic ^ki 1 Is, 

When applieci i^ith forethoughte and preparation, training mi 
reflects thesa premises yields me^aningful results. C>;amples of these 
results are: 1. Increasing commur^ i ca t ion and cooperation bet^isn 
individuals and funct ional grDups^ ^ S, Stimulating objective fePdbac»< and 
a positi^^e f ocui on performance iffmprovement opportunities; 

3. EliminatingdupUcation of effs^rt? Team commitment to cuffifnan g^E^als 

and aChsev^mBnt level si 5. Resolv— ing conflicts and ter r i tor i ai di^pu^=tes ; 
6* Sharpening thi sense of person^^l accountability and job ^dAtiifaC t i^^n ; 
and 7. Clearly identifying strengfc^hs and weaknesses with strategies tcs 
correct per f or/nanc e » 
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Applying thp tot.fli -yBtPm vie-w to haUncing th,. r,fi.„d., c;f p^plp and 
technology qobs beyond the clasaroom and chal lenqos the trsditionfll 
boundariaE- t)fftw.?pn training, operatlonB, and other 1 ine..Euppnrt 
functions. Th^re is no "^ool;booM" to improving worMng rfj] at i onah ip« 
between penple, between functional departmenta. or between people and 
manhines. There are premisea and techniqueci that can be i-iorked into 
P«iating program^,. Thftre are strategies and group lechniquett that can be 
applied to ahaping r ei a t i onah i ps . The i-ey, however, la the individual. 
We cannnt bo "made tc be accountable", or "instructed to think in terms 
of the totaj s/5tem". No one c*n force us, proc-.urfil3y or ntherwise, to 
be empsthetic. These traits are awiahened in people as they see othera 
e;.'hibi^ them in their work, commun i c At i on , tps,, efforts, and in the .,ay 
they thint- and solve problpms. 

The dociBion to invfl^e change in our relationships i-nth other 
people, and the machines, we work with, Is first a choice to permit change 
in ourselves. We must look at the quality of our evperlence decide 
whether or not we are satisfied with the way things are. We can then 
aisess our personal convictiona about the changes that believe ar© 
necessary. From this icwarenesa we can chooBe the ways that we will 
de.Tionstrate our conviction through our actions. 

In the nuciear power industry there is a rigorous demand for 
moment- to-mo^ent awareness of the condition of the total syfiten,. There 
is ^.n !ncreap,ed demand on indi,.inusls to give, receive, and apply 
perfor^anr-,. feedhac^. There .s a need to he consistent), accurate. 
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timely, and complete m th& pprceptiDn and CDmmur-ic-^t: i on nf opr ^ t i an*:^ 
rclat&ri i nf or fTfSt i on , This if. thB field c-f opportunity' fcu= in^lividu«lt 
and organi ra t ions that are committert to efi^ctivf?^ arni prolitahlw 
s t&ady ^3 tat& Dpera t i ons . 

Ideas For AppllcatiDn 

The? folloi^Jing guidelines are bpinn tcsiipn and .^rur i j Dd in fenvCTal 
nunjpar powor plants to d^^-^elop a tot§] Bysto^m vi&w. These quirielm^s- 
emphasi re stratDgipfi that rpinfarc& human learning and ad ju^ tmr^n t . wh i J 
ecGnomizn-.g avsi ' hie rEsaurcrs. 

DeyglQ ainq Teamwork 

Teami%iOT k , frofn iiip total ^vstfDm vicrw. i.ak»?s nn an addpd fi i F-f^n'iA on . 
PeonlF agrpi? that tht5r& is an abp?Qlute nF^cesuity for earhr individual to 
be ^nat^tiedg&atilF and technically cnmpetent in their rolf^. pQr^onal 
accountability, in the conte^^t of the total systems requine^:^ personal 
awarene'ss and discipline that reachee beyond nur tradition^i apprcach to 
teamwori- eind rol^ definition. Thnre is an implind willinGnef-^ to 
identify with somethiing other than thP ?self. and to bri open to o'her^ 
identifying with u?. - gaining knoirjiedae of our s^t renp th^ and weai- nec^ses . 

To hide a wea^'ness or an errnr daffiages the team. De'.elonj^^n 
team^-'jofk involves the trust snri insight to conmunicate ob jnc t i 1 / ^ 
moment-^to -moment 5 ataout the cnnriitinn of tf^e ^ot.^l ^^.y^^tem as each nf 
perceives it. If the p^^evailing culture in an organizational EystefT- i <i 
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punitive, it will r.upprnss the? commun icut ion of facts. For cooDCiration 
to tlQuriah, fact^ must flow unnbstruc i^^id , to and froni the individuals 
that ne^d tf^oB© facta in order to opmratm, 

riost of us have not been trained to recognise that a person is 



psri^ncing extreme stress or overload. How do we notice, in ourselves 



and in others, that the limits of knowledge, endurance, or couraae are 
being reachcrd? Peaple must learn v^hat to look for. What is relevant? 
What IS different in each situation? Developing teamwork, in the total 
system view, relies on se] f-^awar eness and sensitivity to others. 

Trust end confidence in the integrity of the team is vital 
to the accurate assessment of each situation and to mobilizing an 
appropriate responses. Awareness of the trends and tendencies of the 
people and the plant, is essential to the early recognition of problems 
and their cause. 



What is the level of teamwork in our facilities today? Do team 
members actively discuss trends and tendencies? Are difficult decisions 
held off for the ne^t shift to discover and deal with? Are problems ^ith 
individuals masked as technical problems? Do we think in terms of the 
total system, or only within the prescribed limits of our roles? 

Activities to develop teamwork includes 

D Examining the quality of relationship within and between 

functional groups. List a set of critical variables based on 
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the output objectives of the total system. Develop team 
profiles that describe strengths and weakness of each 
f unc t ional uni t - 

o Involving top-management in processes to describe the impact 
and nature of counter productive team tendencies. Include 
individuals at all levels in developing and implementing 
strategies that correct weaknesses and capital iie on strengths. 

© Providing individuals with self development tools to assess 
their technical skills? interpersonal strengthSs and 
nonproductive tendencies* Provide training, counseling and 
on-the-job coaching in the areas indicated, 

o Encouraging the attitude of accountability for team perfDrmance 
by emphasizing individual visibility and specificity of 
objectivess personal choice? alignment of personal values to 
total system objectivesj positive reinforcement of successive 
approximations to the goal. 

ImprpvinQ Cpmmuni^ atiQn Skills 

Different modes of communication are required for various states of 
opBration, The efficiency of communication is measured in terms of 
accuracy, timeliness? completeness, and relevance. The effectiveness of 
communication is measured by comparing the results of communication to 
the intent. When an individual is monitoring plant functions* they are 



110 



III. A. 3. 16 



in a receiving or listening mode. Efficient communication is critica: 
When a response ia indicated, effective communication is critical to 
ensure that the actions taken correBpond directly with lyhat has been 
asked for. 



Training can be designed to focuB on the skills of efficient and 
effective communication. Simulator training provides an evrejleni. 
opportunity to evaluate and test these ski Us. The use of .video cameras 
and playback are being applied to this purpose in several facilities. 



From the total syatem view, all communication between the parts of 
the system are valued. Dperators and superviBors are aware that the 
'•source" of communication is often given an e«tra, judgmental value that 
reflects the receiver's opinion of that source. This results in the 
informational content being down-played or over - emphas 1 sed , dependir.g on 
the positive or negative value assigned to the source by the receiver. 
The receiver is often unaware that they are applying this bias to the 
information. Source bias results in the degradation of the arcuracy r.nd 
effectiveness of communication. Sel f-assessment tools that msasure 
response bias in communication can be applied in bo^h formal training and 
coaching activities. Re=qua 1 i f i ca t i on training is an r.>.cellent 
opportunity for operators to explore habits and communication patterns 
that may be blocking valuable information on the job. 

We are often trained in the basic mechanics of cnmmun i ca t ion ; who 
talkr to whom, when. We receive littlr- or no training regarding the 
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impact that we, a& individuals* have- on th© c {mmuni cat ion process. 
Training that empnasires process stills and personal ai^areness cf the 
subtleties and r pspons i b i 1 i t i ©s of communication is necessary. Technical 
instructors^ train&d in these techniques^ are able to reinforce positive 
communication patterns in students in all of their regular course work* 

Assessing communication needs and improvements can be accomplished 



o Identifying the critical modes of operation and the 

communication requirements for each. Specify the criteria for 
measuring efficienc/ and effectiveness* Reinforce these 
criteria during regular course work and simulator training* 

O Providing individuals with "process" training that demonstrates 
the impact of the individual on the results of communication* 
Teach instructors, senior operators* and supervisors to coach 
and reinforce positive communication patterns* 

o Providing of f-^ line practice and coaching for plant situations 
that are known to occur frequently* Skill practice should 
mirror the individual's actual job es'perience* Deal 
specifically with the impact of positive and negative "source" 
bias? anger? boredom, apathy? and panic* Use video feedback 
and group discussion techniques to encourage feedbaci^ betwef?n 
par t icipants* 




III. A. 3, 18 



s Emphasizing communication efficiBncy during simuUtor training. 
Show that the way in which w© request information affects the 
quality and accuracy of response, 

Q Including operations supervisors in on-thB^job reinforcement 
and coaching strategies to insure transfer of skills. 

Thinking and Diag^omtic Bkillm Training 

Research indicates that human intervention in plant events during 
the first thirty seconds increases the probability that the situation 
will be worseneds by a factor of one. 

Simulator instructors have observed that one of the primary causes 
of test failure is the tendency to react too quickly i before getting a 
"big picture". Several operations managers believe that many system 
trips are caused by people over-operating the plant! not taking time to 
observe the situation and to think. When an individual is trained to 
respond to specific indications and annunciators, and 15 evaluated 
comparatively on the basis of mean-t i me-to-response 5 certain affects are 
predictable* The individual will be acutely sensitized to perceive 
information specifically associated with their station, and they will be 
highly motivated to respond quickly. 

These response patterns are a direct reflection of hot*i we train 
individuals to obtain a license. We teach people "what to do" as opposed 
to "how to think". 
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Converselys when an individual thjnks from th© total system view, 
they are thinking as a member of the team. When we teach memorization 
end test only retention, we are encouraging peaple to think of themselves 



individually, and there is a definite efficiency in traditional modes of 
instruction. There is also the probability? that the exclusive use of 
these methodSf has a dulling effect on the indi v/idual 's ability to think 
globally. It is important to balance instructional methods to stimulate 
thinhing and the ability to visualize what is going on in the plant. 

Strategies to help balance the instructional approach and to develop 
thin, ing skills arei 

O Teaching people how to learn efficiently at the outset of 
license and certification cnurses. Instruction in various 
memory and recall strategies will assist people in acquiring 
fundamentals faster, and in preparing for applications and 
simulator training, 

Q Training instructors m techniques that cause highei levels of 
integration and association in the learner. Using case 
problems in conjunction with group process terhniques is an 
effective method of drswii^g out thinking and comrnuni ca t i on 
skills. Having students analyne and interpret s i tc?^5pen i f i c 
event reports builds confidence and analytical skills. 

o ImprDving methods for evaluating instructor performance in 



in isol at ion. 



There 



obvious need to qualify and license people 




ni.A.3.20 



simulator and lab ©nv i r onment s . Use student feedback and 
"round-table" discussions to learn, from the student 
perspective, how the resources in these environments are being 
utilized by instructors. Brainstorm ideas for improvement with 
the students ^ 

Q Providing training in visualization technique and the use of 
mental discipline to overcoine emotionality, stress? and 
inappropriate reflexive responses. Give students problems with 
simple solutionsp under high-^stress conditions. Train them to 
''think through" emotional and physical reactions by practicing 
perceptual disciplines, objective assessment of the "facts'*, 
and "at station" rela>?ation techniques. Tie to skiiis of 
communication efficiency. 

o Providing training and reinforcement in the adoption of a 
rational model for problem solving and trouble shooting. 
Develop efficiency by using the same model plant wide. 



Q 



Providing opportunities in license and re--qual i f icat i on 
training to learn and practice approachss to team problem 
solving and trouble shooting. 



Licensed and unlicensed personnel should be introduced to tBamwork, 
communicat ionp and diagnostics skills, using methods that will help them 
manag© their relationships in the work environment. By optimizing the 
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timing, deliver/j and reinforcement of certain principles^ t ainers can 
inject attitudes of teamwork into existing hot-license classeBs 
re-qual if ications and maintenance training. 

Principles of teamwork can be included as a standard introduction to 
System Fundamentals» emphasizing the attitudes of accountability and 
empathy for the total system. These principles can be successively 
reinforced in subsequent Fundamentals Modules that teach procedures and 
mechanica) repair skills that are team dependent. 

The standard instruction of system diagnosticSf trouble shootings 
and plant maintenance will be enhanced by the introduction of thinking 
skills and rational problem solving techniques. All groups that receive 
specific skills training in diagnosing and correcting plant systemSi can 
be simultaneously exposed to critical thinking skills* 

Utiliiing the simulator during re-qual if Icat ion training, senior 
operators and supervisors should be exposed to coaching skills that can 
be applied to their regular shift duties. When license candidates are 
assigned to a unit for Q.J,T,f the senior staff and supervisors will be 
using the same techniques that the trainee's simulator instructors will 
use in the next phase of their license course, 

A LEADERSHIP ROLE FOR TRAINING IN STEADY-STATE OPERATIONS 

The challenge of providing leadership is not limited to those with 
management authority. Each functional group can contribute to 
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establishing direction and positive momentum. Training, as a functional 
discipline, has mult, le opportunities to introduce attitudes and valueE 
that have a positivB influence on total plant operatiDnB. Introducing 
theBB attitudBB requires inBtructors and training managerB to lead by 
personal eHample. 

Many technical training functions have a skeptical attitude 
regarding change. Alterations to the core curricula require ma-iaive 
efforts from staffs that are alreadv buried. Most technical training 
profBBBionalB are reluctant to make sweeping changem for fear of 
"throwing the baby out with the bath water." What is suggested is not a 
replacement or reconstruction of content, but rather an infusion of the 
total syBtem view into a base of training that already works. The 
training function can assume a leaderBhip role in the introduction and 
reinforcement of concepts that support teamwork, communication, 
leaderBhip, and diagnosticB. These skillB and attitudes have a unifying 
influence, when applied from the total system view. 

Chal IsngeB 

Several chaUengeB need to be met for Training to effectively assume 
a leadership role; 

O Eliminate the terr i tor i a 1 i sm and in=fighting that evists 
between most training departments, 

o E>!pand the basis for on-going collaboration between training 
and operations groups by establishing cooperative objectives. 
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O Identity the i mprovempnts needed 3n in^tr^J^tfM= skill- tr 

erf&ctively rnDdPl snfl emulate tho attittjd&i. dt'Sirt-d in riifit>r£. 

D Align the? strategic vision for training v-ath thr= stt^aiegy of 
nD&ratinnEp I nvDlve Dpe-rat ionfi manaQem»ni and licenf:.Dd E.taf=f 
in a team approach to planning^! curticula dFcign? 
iTjplementat ion, evaluation and r i n+'or ce,t.Rni of trair,ing 
sy s t sms * 

O Apply learning strategieii that aro participant centered. Tearh 
methods that reduce the learning curve in area^ that are memory 
and recall i n t ens i ve . 

D Integrate process concepts into the e^tablinheri sruueriC e of 
training fn^ licensed and nnn-licensed personnel. 

o Adopt coaching and facilitation skills that will improve the 
perfDrmar.ee of simulator instructors. 

CONCLUSION 

Each indi^/idual must consider their role and respon*^e to integrating 
the peopJe and techno Jog y m their operation. llhen wo adopt the attitude 
of accQuntab i 1 i t y ? our specific role and contribution taieu on a hroader 
scope, allowing more personal sat i rLfac 1 1 on . 



As the environment matures- we are swept dally into the unknown* 
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Mhen we .acnoun t jh i U t / , ^.rs mati,.a.^d i;o !,iok for ways to 

impr-,vw our performance, our r e 1 a t i o , ,Sh i p ; , and aur i&ility to ps^rsunal 
Btf,frt; n„oiii.B liurcoms Tor ths o-ntire ar-sr^tinn. 

How tho nuclBar power induBtry perfurms o ^tsr the t thirtv to 
mivty vears will have am much or little to do with us as w,> choose. tf 
you or ! stand up dnd declare, "I am accountable for this industry, the 
technoloijy. the people, the training, and I will, by prBsence and 
choics, have a positive effect on tne outcome!"... Hho wDuid argue' 
Would it make a diffore^nceT rhe choices, as I see them, are to work 
consciouBly to improve the odds far buccbss . or to sit on the edge of 
change and wa i t . 
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ABSTRACT 

The Niiciear Regul£itory Commisuion (NkC) fias had a policy^ 
Generic Letter BZ-IZ, on hours of vvork since The 
pQl1::y states that licensees should astabUsh controls to 
prevint situations where fatigue could reduce the ability 
of operating personnel to perform their duties safely 
(USNRC 1982). While that policy does give guidance on 
hours of work and overtime^ it does not address periods 
of longer than 7 days or work schedules other than the 
routine 8-hour day, 40-hour week, Recoynizing that NRC 
policy could provide broader guidance for shift schedules 
and hours of overtinje work 5 the Division of Human Factors 
Safety conductid a project with Pacific Northwest 
Laboratories (PNL) to help the NRC better understand the 
human factors principles and issues concerning hours of 
work so that the NRC could consider updating their policy 
as necessary. 

The results of this project are recominendations for 
guidelines and limits for periods of 14 days, 28 days, 
and 1 year to take into account the cumulative effects of 
fatigue. In addition, routine 12-hour shifts are 
addressed. This latter type of shift schedule has been 
widely adopted in the petroleum and chemical industries 
and several utilities operating nuclear power plants have 
adopted it as well. Since this is the case, it 1s 
important to consider including guidelines for 
implementing this type of schedule. 

This paper discusses the bases for the PNL 
recommendations which are currentlv beino studied by the 
NRC. 

This project is one example of how human factors 
specialists can help trainers, workers, and licensees 
achieve improved human performance by providing rationale 
for appropriate work schedules. 
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IMhODUCTION 

Since ISBZi NRC po11f,> on hours of work in the nuclear power industry 
hb$ consisted of the guicJance in Generic Letter 82-12. While thrt 
policy addressed hours of work and overtime, it only dr-alt with 
periods of up to 7 days and work schedules of routine Q-hour days and 
40-hour weeks. Routine 12-hour shifts were not considered. This 
lattf-r type of shift schedule has been widely adopted in the 
petroieum and chemical industries and several utilities operating 
nuclear power plants have adopted routine 12-hour shifts. While 
there is nothing in present guidance or regulation against such 
shifts, neither are there any gufdelines for implementing this type 
of schedule. 

Recognizing that NRC policy could provide broader guidance for shift 
schedules and overtime worK, the Division of Human Factors 
Safety, in cohjunction with Pacific Northwest Laboratories (PNL), 
conducted a project to help the NRC better understand the issues 
concerning hours of work so that the NRC could consider updating 
their policy as necessary. 

PROJECT TASKS 

During the early stages of the project, the PNL project staff 
produced a critical review of the literature nn shift scheduling and 
overtime. It was orlginciily proposed that the Initial phase would be 
a review and evaluation of the available literature pertaining to 
shift scheduling and overtime practices in nuclear power plants. It 
soon became evident that little or no research had been conducted in 
these plants. The solution wns to review research in nonnuclear 
industries that have similarities to thf nuclear power industry. The 
study Included federally regulated industries, the military, and a 
number of field and laboratory studies conducted on hours of work and 
performance. 
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The second major task of the project was to assenible a panel of 
experts to consider limits on total hours of work. Panel members 
were nine well-known resear*chers ^ medical doctors, physiologists, 
psychologists 5 and administrators whose experience spanned the Navys 
Air Force, and the CQast Guard, and the commercial airline, railroads 
and petro=chem1cal industries. All the members of the panel were 
selected because of their professional concerns in the area of 
fatigue and human performance. During a 2=day v/orkshop, the panel 
developed suggested limits on total hours of work for both 8-hour and 
12»hour daily shift schedules for weekly ^ biweekly, monthly^ and 
annual work periods. 

The final phase of the project was to provide recommendations to the 
NRC for new or modified guidelines on shift scheduling and hours of 
work for nuclear power plant personnel who perform safety=related 
work 1n nuclear power plants. The recommendations covered four 
aspects of shift scheduling and hours of work: 

(1) limits on hours of work (including overtime) 

(2) routine 8-hour/day shift schedules 

(3) routine 12=hour/day shift schedules 

(4) total number of control room operators at a plant. 

SOURCES OF DATA 

The primary sources of data that were used as technical bases for the 
recommendations were: 

^ field and experimental studies in nonnuclear occupations, i,e,s 
airline pilots, truck drivers, railroad operators, air traffic 
controllers; laboratory experiments on reading speed, vigilance^ 
mathematical abiHtyi scores on a variety of cognitive tests* 
Most of these studies were reviewed in the project's first 
report (Lewis 1985 a). 
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the experience and knowUdge of tho pancil of experts assembled 
as part of this project, 

fatigue indexes that were dnveioped fur airline pilots, (A 
fatigue 1nde>: is an alporlthm that estinintes the level of 
fatigue or perforninnre based on factors thnt cause fatigue, 
especially hours of work,) 

Interviews with employees In the nuclear industry and in 
nonnuclc^ar industries . 

In general, the recommendtitlons are baf^ed on a judgmental evaluation 
of accumulated evidence from many sources, rather than on a single 
piece of evidence (Lewis 1985 b). This approach was influenced by 
the fexperlencp of other branches of the federal government that have 
also studied feticiue* 

RECOHMnNnATIONS 
Hours of Uork 

The recommendations cnncernino limits on hours of work were actually 
built on the limits included in current NPC policy. The difference 
lies in that the recommendations were designed to take into 
ccrsideration the cumulatHe effects of fatigue. Simply put, you may 
be able to work extraordinarny long hours far a week or even two^ 
but if you continue to do that for a month or two or six, you will 
begin to feel increasingly fatigued. Consequently , the 
recomrnendatirns on hours of v/ork include guidelines for 14 days, 20 
days, and ] year. 
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?'ficommendf:fj Pf'licy on Hours of Work 







Utility Must at 




r laMt l-C3fiGMEI 


i cj ii t ^' T n 




Apprcval Required 


NPC if These 




to ExceBd Thesp 


Linits are to 


I'irjife F*priod 


L i rn 1 1 s 




1 day 


12 (16)* 




2 days 


£4 




7 days 


60 


72 


14 days 




132 


ZB days 


192 


2?e 


1 ymr 


2260 


23G0 



*If an operator 1s absents his rfplacement mty work up to 16 hours, 

Tht- recQinmend^tlons 1n the box in th^ table above are those inciudad 
In current policy v/lth thr exception of the 12-hour llniit in 1 day. 
Present NRC policy allows an Individual to work up to 16 hours in 1 
day, excluding shift turnover time. The recomniendatlnn for a 
12-hour limit in 1 day is baperi on a nurnb^iir of sources, numerous 
reports on nonnuclear industries Indicate that fatigue incrpe?es 
after P hours and increases rapidly after 12 hours. Both present dr.d 
reconimended limits are less stringent than In other regulated 
industries^ I.e., airline pilots and truckers. It is pcssible, 
however, that the riifferencu can be Justified because the nuclear' 
industry has more backup safety system?. Experlmenl^l research shows 
performance decrements In a variety nf cognitive and psychomotor 
tasks. For instance, reading speed slows aftur 4 hours and. after C 
hours, pilots' performance In flight simulators declines, and error^; 
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on CDinplex tasks Increase. In actudl on-the-job situations, after 8 
hours s aircraft accidents and occupational injiiries Increase and 
drivers take greater risks. Observations of flight crews indicate 
that after 12 hours, pilots become more Irritable^ less 
comrnunicativet take more riskss neglect more factors, and forget 
more (Lewis, 198S a). 

The bases for the limits on the remaining time period are the 
recommendations of the expert panel, various fatigue indexes, and 
limits for other regulated occupations. In addition, a 2-year study 
by one nuclear utility shows that productivity "drastically 
decreases" after 72 hours, i,e., 6 consecutive days of 12-hour 
shifts. If productivity decreases drastically, the rationale for 
working such long hours decreases drastically along with the quality 
of the work. 

Recommended Guidelines for Routine 8-Hour/Day Schedules 

Since the majority of nuclear power plants still maintain routine 
8-hour/day schedules, It 1s important that they be implemented in 
such a way as to minimize fatigue. For this reason, PNL made 
recummendations for this type of shift, as follows: 

1* The schedule should be limited to a maximum of 7 consecutive days 
of work* 

2. The schedule should not exceed 21 days of work (Including 
training) in any 4-week period, 

3. The schedule should include at least 2 consecutive full days off 
in any period of 9 consecutive days. 

4. A series of night shifts should be followed by at least 2 full 
days of rest. 
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5. The schedule should rotate forward, not backward. 

Once agains the bases for these recomrnendations are the panel's 
expertise and a great deal of existing research on fatigue, especiany 
as it relates to rotating shifts. Evidence indicates that fatigue 
accumulates toward the end of a long series of work days. The 21-day 
limit in a 4-week period follov/s the same reasoning as does the 
recommenddtion that there be at least 2 consecutive full days off in 9 
consecutive days. Two full days 1n this context means 64 hours s which 
Is the number of hours in ^ normal weekend. Two full days off are 
recommended after 6 or 7 consecutive night shifts because this shift 
disturbs one's circadian rhythms most, (Circadian rhythms are bodily 
rhythms that oscillate within a period of approximately one day. Not 
only is there a sleep/wake cycle, but body temperature^ gastric 
secretions ^ and many other bodily functions have circadian rhythms.) 
When on the night shift, one works when one v/ould normally sleep. In 
addition, people have difficulty sleeping during the day, so a sleep 
deficit occurs. Therefore^ a longer rest period Is advisable after 
the night shift to readjust one's circadian rhythms to day work and 
night sleep, and to make up the sleep deficit. Forward rotation, 
I.e,, days to evenings to night, theoretically allows faster 
adjustment of circadian rhythms 1f (1) the worker anticipates the 
next shift and begins to adjust his sleep/wake cycle before rotation, 
and (2) the worker maintains the sleep/wake cycle appropriate for 
work even on days off. There also needs to be sufficient time off 
between rotations. If these conditions are not met, a forward 
rotation may fail to reduce fatigue. It should be noted that the 
benefits of forward rotation are still being debated by researchers. 
However, a number of operators interviewed stated that they found 
forward rotation less fatiguing. 
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Recommended Guidelines for Routine Ig-Hour/Day Schedules 

It is understood that many utilities adopt 12-hour/day schfidules when 
overtime is expected to continue for a period of time. This 1s not 
the type of schedule to which these reccmmendatlons apply. Routine 
12-hour schedules are generally designed so that the work week still 
averages 40 hours. It also usually entails renegotiation of the 
basic wage rate so that base pay plus "overtime pay" still yields the 
same total salary as did the routine 8-hour/day schedule. As 
previously mentioned, there is nothing in current NRC policy that 
precludes routine 12-hour shifts; neither are there any guidelines 
for im.plementlng such schedules. This type of schedule has already 
been adopted by one utility and other plants are considering it. 
With this in mind, PNL made the following recommendations: 

1. The schedule should contain a maximum of 4 consecutive 12-hnur 
work day: . 

2. Four consecutive 12-hour work days should be followed by no 
fev/er than 4 days off. 

3. The basic 12-hour/day schedule should be "Z-on, 2-off," "3-on, 
3-off," "4-on, 4-off," or a systematic combination of these such 
as the "every-other-weekend-off" schedule, which combines "2-on, 
2-off" with "3-on, 3-off." 

4. The general safety record of the plant should be satisfactory, 
based on criteria such as those used in NRC's Systematic 
Assessment of Licensee Performance (5ALP) ratings. 

5. The plant should have the ehillty to cover unexpected absences 
satisfactorily without having any individual work more than 12 
hours per day. 
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6, The round-trip comute times for the operators shoLAld not exceec^ 
M/2 hours. 

The first three recoisirnendfitlons are based on existing 12-hour/ day 
schedules. One nuclear plant adapted a "4-ons 2-off" schedule, but 
later abandoned it because operators became too fatigued. Although 
the bulk of evidence collectari so far indicates that 12-hour shifts 
are safe and raise employee moral e, they are relatively new and 
unused compared to 8— hour schedul es . Until NRC and the nuclear 
Industry gain more experience v/1th this shift schedule, some 
consideration certair^ly needs to be given to the overall safety 
rGcord of the plant* The fifth recommendation is necessary because 
you cannot fall hack on having someone work a double shift (24 hours Ij 
to cover for an unexpected absence. Possible solutions are to 
maintain people on cs 1 1 or staff each crew with eKtra operators. 

The last recommendat* on was Included because excessive comrriute time 
inherently leaves less time for sleep. European researchers Include 
adequate sleep tifne as a criterion for the acceptability of a 
li-hour/day shift scf^edule. 

It should be noted th^at since the completion of this project, PNL 
conducted a study of the 12-hour shift schedule at the Fast Flux Tfex& 
Facility (FFTF) at H& nford^ Wtishington. (The 12^hour schedule used 
at FFTF was as follov^s: 4 nights; 3 off; A days; 7 off; 3 days; 4 
8-hour training daysi. 3 off; repeat,) The schedule was changed 
from ci routine S=hour^/day shift schedule with the primary objectives 
of reducing attrition and Increasing job satisfactlcn , Because of a 
concern that potently 1 fatigue could lead to errors that would 
jeopardize plant saf&tyjthe effect of the 12-hour shift on safety 
was assessed by coniptL ring the number and severity of nff^normal 
events on the 0- and 124our shifts- The result of this analysis 
Indicates that there v/as no sstisfactori ly nieasurable difference 
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between the 8- and 12-hour shifts in either the number or severity 
events. Error rates in keeping Technical Specification Compliance 
Logs was lower on the IS-hour shift. Operators definitely were 
somawhat more fatigued on the 12-hour shift , but this 1-year triaT 
apparently did not l^ad to any sacrifice of plant performance or 
safety (Lewis et al , 1986). 

The final aspect of the recommendations deal t with the total nuniber 
of control room operators at the plant, Although PNL considered 
reeormiending that each plant have enough control room operators to 
staff six shift crews, this idea was rejected on the basis that 
setting limits on hours of work Is a more direct way to deal with 
overtime and fatigue. A recoimandation was made that NRC collect 
data on the number of control room opirators currently on staff at 
nuclear plants in order to assess the degree to which understaffi 
may be an underlying cause of overtime in the industry. 

Conclusion 

Shift work, fatigue, overtime, and performance are hardly new issues 
Because we initially found little work in this area done In the 
nuclear power industry does not mean it is irrelevant to us. In IgS 
and 1982, the NRC and PNL held a series of workshops as a mechanistTt 
for obtaining feedback from power plant operators. During these 
workshops, operators expressed concern that fatigue could have safet 
implications (McGuire, Walsh, and Boegel, 1984). In a subsequent 
survey carried out as an alternative feedback mechanism, over half 
the operators surveyed said that overtime work sometimes creates 
plant safety problems. That survey coyered 520 personnel at 26 
nuclear power plants (McGuire, Walsh, and Morisseau, 1985). 

This study was designed to bring together the best comprehensive 
human factors knowledge available and use it as a technical basis W 
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whatever changes might be necessary to current NRC policy* These 
recorrwOTendations are still being studied to determl ne which are 
mana^geable or feasiblef while also taking intocons ideration the 
prodi^ctivi ty and safety of nuclear Instanatlons 1 n the United 
States. 
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THE TTRAINING DEPARTMENT'S ROLE IN HUMAN FACTOR ANALYSIS DURING 

POS T-TRIP R EVIEWS 



Deborah Goodmans Ph.D. 



ABSTRACT 

"Provide training'^ is a frequent corrective action 
specified in a post-trip review report* This corrective 
action is most often decided upon by technical and opera- 
tio^ial staff, not training staffs without a detailed analy- 
sis of whether training can resolve the immediate problem 
or ^^nhance employees' future performance, A more specific 
hume.n factor or performance problem analysis would often 
rev^sal that training cannot impact or resolve the concern 
to ^avoid future occurrences. This human factor analysis is 
similar to Thomas Gilbert's "Behavior Engineering Model" 
( Hu^nan Competence s McGraw-Hill , 1978) or Robert Mager^s/ 
Pet^r Pipe's "Performance Analysis" ( Analyzing Performance 
Pro^^lems , Pitman Learningt 1984)* At Palo Verde Nuclear 
Gen^^rating Station^ training analysts participate in post- 
tri^ reviews in order to conduct or provide input to this 
typ^e of human factor and performance problem analysis* 
Their goal is to keep "provide training" out of corrective 
action statements unless training can in fact impact or re- 
sol've the problem* The analysts follow a "plant specific" 
logic diagram to identify human factors and to identify 
whether changes to the environment or to the person would 
beafc resolve the concern. 



INTRODUCTION 

Wheo. a plant trips, analyzing root causes, determining corrective 
actionss and bringing the plant back on line are paramount concerns to 
the opera, ting staff* The performance data from systems and components 
and the ©ports of actions and observations by operators and support 
personnel are analyzed to identify areas of concern* Questions are 
raised as to how these concerns may have served as potential causes or 
aiding fa^ctors in the trip, and research is conducted to prove or dis- 
prove tha^t the identified concern is part of the root cause* As this 
type of ^view progresses , concerns become findings, and recommendations 
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for corrective action are mad^e. Quite often, part of the corrective ac- 
tion to many findings is to pteTovide training--to place the item in the 
operators' next requalif icatl^n cycle, to embed it in initial training 
lesson plans. 

Many of these training recommendations are not appropriate correc- 
tive actions. A more specifics human factor or performance problem 
analysis would reveal that training is not the solution to avoid future 
occurrences. Models of this t=ype of analysis can be found in Robert 
Mager's/Peter Pipe's "Perf ormftsnce Analysis"^ or Thomas Gilbert's 
"Behavior Engineering Model''^ 

MAGER: PERFO' C3RMANCE PROBLEM ANALYSIS 

Mager defines a perforniBn = .ce problem as an instance where people do 
not do what they are supposed - to do or what some other person wants them 
to do, and these problems are mnade evident by statements such as: 

"They don't have the righ«-t attitude." 

"We need a course to teacEih people..." 

"We've got a training promjlem because our workers aren't safety 
conscious. "(1, p.l) 

Responding to such statements with training will help sometimes: if 
people do not kno« how to perform, instruction is likely to help. How- 
ever, when people have known Hq=w to perform at one time or another, 
teaching or telling them agairi is not likely to remedy the problem. 

If the performnce probleniEi really needs to be resolved, the first 
question to ask inMager's anal ^ysis is whether the non-performance is 
based on a skill deficiency. 1= n other words, can people do what la 
expected of them if their lives depend on it? If they cannot do what 
is expected of theni, thare is a skill deficiency and further analysis 
related to the need for traintn*^ should he pursued. If people could 
perform in the past, but not nou^, the analysis should determine if they 
perform infrequently and nRed pitractice or if they perform frequently but 
need feedback on correct perf oriKTiance. The analysis should also deter= 
mine if the performnce is too c— omplex and results would be better 
achiovod by providing Job aids, procedures, greater on=-the-Job training. 
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or actual work simplification* And if people have never beeTi able to 
perform as expected, the analysis should show the need for formal 
training followed by practice and feedback. 

In Mager's analysis j if people can do what is expected of tliem "if 
their lives depend on it,-' a skill deficiency does not ©Kist * Tlie per= 
formance problem analysis should focus on other human factDrs as causes 
of the problem, with four courses generally examined i 

1* People don't do what is expected because it is punishing to 
perform, 

2. ...because it is rewarding not to perfornii 

3. because it does not matter if they door don't perfor^m* 

4. because there are obstacles to perforniing, (If p» 3 5 

GILBERT! BEHAVIOR ENGINEERING MODEL 

The external environment and the person— matrixed against a grid 
of Information, Instrumentation * and Motivation^--are the interacting 
areas which must be analyzed to solve performance problems using Gil- 
bert's Behavior Engineering Model (Figure 1), 
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Figure 1* Behavior Engineering Model (2, pp. 83--88> 
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Analysis of the Environment/Information cell calls for identify^ 
^r^g whether people have the data they need to perform wall.# This data 
.nr^cludes identification of expected performance, clear guides on how to 

e^complish performance, identification of standards associated with per- 
fc^rmance, and relevant and frequent feedback regarding adequacy in meet- 
...= 8 standards of performance. Analysis of the Person/Information cell 
ea .lis for identifying the skills and knowledges people will need to use 
or implement the data from the environment. 

For the Environment/Instrumentation cell, the analysis should 
Identify whether there are adequate and accessible references (proced= 
ur^s and persons), tools, and equipment; correct data with which to 
wot=^k; problem-solving authority; satisfactory work facility; and ade= 
qu^te work design and supervision. For the Person/Instrumentation cell 
th^ analysis should identify whether people have the capacity to perform 
th^ actual task; to learn the skills and knowledges needed to perform- 
and_ to use the tools, equipment, references and other things available 
to them m the environment* 

Finally, for the Environment/Motivation cell, the analysis should 
Identify whether there are meaningful incentives and rewards for per- 
for.=^xng as desired-^incentives such as recognition, compensation, 
autonomy, and advancement. Analysis of the Person/Motivation cell 
woulid identify whether people have expectations or motivations which 
are compatible with the available incentives and which would lead them 
to perform as desired in order to achieve the incentives. 

THE TRAINING DEPARTMENT'S ROLE IN HUMAN FACTOR OR PERFORMANCE 

PROBLEM ANALYSIS DURING POST-TRIP REVIEWS 
Given the technical background and orientation of most individuals 
assigned to post-^trlp reviews, analysis of the performance problem aa= 
soca^ted with the trip focuses on Gilbert's Environment/Data and Environ^ 
manWlnstrumentation cells. Procedures are reviewed to determine 



-(Discussion of the cells includes Donald H. Bullock's amplification in 

UntEmerstanding Gilbert's Human Competence . "^^ 
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adequacy and accuracy for guiding perforniance and equipment/system fail- 
ures and malfunctions are analyzed for their root cause contribution. 
Problems with the performance of people are most often assumed to be a 
"skill deficiency-- (Mager) or a ''lack of knowledge" (Gilbert), 

Many times these assumptions are correct* the person could have 
performed better as a result of training, practice, feedback, or infor- 
mation—the potential solutions to a skill deficiency. Quite often, 
however, these assumptions are not well founded, and commitments are 
made to provide training which is inefficient and ineffective for re- 
solving the problem. Equally often, the Training Department has no in- 
volvement in making these commitments and sometimes even has no knowledge 
that the commitment has been made~only knowledge that it has not been 
met. 

To avoid these inefficient and ineffective commitments for trainings 
the Training Department should have an active role in identifying whether 
a skill deficiency contributed to the performance problem and should 
have responsibility for recommending the most effective and efficient 
method for correcting the skill deficiency if one is found. Mager 's 
"Performance Problem Analysis" and Gilbert's "Behavior Engineering Model" 
represent appropriate analysis tools. In both Mager 's and Gilbert's 
approach to performance problem analysis , factors which can contribute 
to a training need are presented first—the analyses determine whether 
people have the information, skillSs knowledge, and practice necessary 
to perform. If these factors are present or do not account for the per- 
formance problem, then the analyses consider other human factors which 
training cannot affect to any significant degree* This is an important 
sequence for Training Department personnel to follow, as it provides 
informatior and feedback on necessary improvements to training or 
clearly identifies that training cannot resolve the problem. The need 
for analysis of other human factors can be identified and, if the 
Training Department has the cooperation or authorlEation to examine 
these factors, the analysis can proceed. If not, a skill deficiency 
analysis can still be completed without stepping into sensitive areas. 
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PVNGS EXPERIENCE 

At Palo Verde Nuclear Generating Station, training analysts with 
instructional design and occupational analysis backgrounds participate 
in post-trip reviews. Every effort is made to have an analyst working 
with the Duty Shift Technical Advisor at the time technical data has 
been compiled to describe the events of the trip and rough concerns have 
been drafted. The training analyst works with the STA to assimilate the 
actual performance problem; discusses and resolves any obviously mis- 
directed training recommendations or concerns; then researches, analyzes, 
and resolves any remaining training concerns. This requires close in- 
teraction with training instructors and supervisors to ensure an accu- 
rate assessment of the problem and appropriate Training Department 
response. 

The training analysts' participation in post-trip reviews was a 
direct response to the problem of having training commitments made by 
external organizations without the Training Department's input and 
without adequate analysis to determine the appropriateness of their 
commitments. Through participation in trips which occurred as Units 1 
and 2 went through power ascension tests and into commercian operation, 
the analysts and the Training Manager assembled an analysis logic 
and flowpath for post-trip reviews. Since the goal of participation 
was to avoid inappropriate commitments, the analysis flowpath began 
with investigation of human factors other than skills deficiencies as 
possible contributors to the performance problem. The last factors 
analyzed were whether the person possessed the skills and knowledges 
needed to perform and whether the training program adequately presented 
those skills and knowledges. 

As the training analysts participated in more reviews and as the 
process became more and more formalized, an internal concern was raised 
about the appropriateness of investigating non^training human factors, 
the potential for the analysts' internal and informal records to become 
part of the official post^trip review, and, as a result of that escala- 
tion, the need for having people sign interview notes to document their 
responses to the analysts' questions. 
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These concerns led to a reevaluation of the analysis flowpath and 
a review of human factor and performance problem analysis literature 
(Mager and Gilbert particularly) , The following analysis 
questionnaire and flowpath were developed to focus clearly on 
identification of a skill or knowledge deficiency and the need for 
training to correct that deficiency. This restructuring allows the 
Training Department to carry out its appropriate role in human factor 
analysis during post-trip reviews, 

PVNGS TRAINING DEPARTT^ffiNT POST^TRIP REVIEW QUESTIONNAIRE 

1. Summary of Performance Problem (What was done which should not have 
been done? What was not done which should have been done?) 

2. Was adequate data available to guide performance? (Do procedures 
or Job aids exist to cover performance? Do the procedures or job 
aids provide corrects clears &nd understandable directions on per- 
formance? Are standards of performance clearly stated in the pro- 
cedures or Job aids? Have standards of performance been communi- 
cated well by other means?) 

3- Was the available data used correctly? (Were procedures follQwed?) 

4, Identify the skills and knowledges the person would need in order 
to use the data or perform the task* 

5* Are the skills and knowledges appropriate content for the training 
program? 

6# Have the skills and knowledges been included in the training pro- 
gram? (Verify and list course(s) and lesson plan number(s) for 
initial and continuing classrooms simulator j and on--the-job train- 
ing as appropriate. List plant specific task numbers when possible 

7. Is the coverage of skills and knowledges of sufficient depth to 
support use of data or performance of the task? 

8. Did the person participat in the training/retraining? 

9. Has the person forgotten to use the data or perform the task? 
(How long has it been si> the person participated in training/ 
retraining? Has the perso.. applied the skill or knowledges used 
the data, and performed the task since training? Has the person 
gotten feedback on using the data or performing the task?) 

10. Do the responses to questions 8-9 describe 

None/few 

Many/mos t 

of the other persons who are also responsible to use the data or 
perform the task? 
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PVNGS TRAINING DEPARTMENT POST-TRIP REVIEW ANALYSIS FLOWPATH 

(Continued) 




^--s^u f f 1 Q 1 © n 1 1 y 


R@QQmmdnd 
ImprDva 
Training 










Resommend 
Corre Qtlve 
Aotlon for 
Individual 



Yes 




RooonriiTiand 
Oorrdatlve 
Aatlon for 
Individual 



139 



III. A. 5. 10 



PVNGS TRAINING DEPARTMENT POST^TRIP REVIE-,' ANALYSIS FLOWPATH 

(Continued) 
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HUMAN FACTORS & TRAINING 
THE PARTNERSHIP AGREEMENT 

A,C* Macffis 
A,C* Macris Prrofessional Consultants 

S * T * Fleming 
Northeast Utilities 
Training Development Supe '^-^ l i 

INTRODUCTION 

Four fundamental activities directly affect human 
performanca in operating nuclear power plants i 

Control Room Design Reviews (CRDR's) 
Operating Procedures 
Training Curriculum Materials 
Simulator Training 

Typically it was believed that mul ti=disciplined 
core teams/ for each activity/ provided an integration 
of all activities* Representatives of each discipline 
(CRDR/ Engineering/ Training/ Simulator Project) provided 
real time inputs during team deliberations* While these 
inputs affected team decisions/ there were no assurances 
that any functional follow-up would result* Futhermore/ 
no mechanism existed for systematic integration between 
activities- Now/ with a majority of the Control Room 
Design Reviews complete/ plant specific simulators becoming 
a reality/ and the incorporation of Safety Parameter 
Display System (SPDS) and Symptom Based EOP ' s ; the reality 
is that these activities require more systematic integration 
than was previously recognized* 

This paper presents an innovative approach for 
integrating the above four activities using Computer 
Aided Drafting (CAD) and computerised Data Base Manage- 
ment (DBM) to synergistically optimize human performance. 
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ACTIVITIES AFFECTING HUMAN PERFORMANCE 
With increased awareness regairding the need for 
improved, human performance, an attempt to more clearly 
define the substantive interrelationships ia being 
made. Each of the four activities affecting human per- 
formanca are discussed indicating the interrelationships 
from our experience and perspective. 

Control Room Design Reviews 
The primary output of the CRDR is validated control 
room design improvements which meet good human engineering 
practices, and have the •stamp of approval' from the 
Operations Department. The control panel configuration 
subsequent to the CRDR has a direct effect on the other 
three activities. The magnitude of design modifications 
will dictate the impact. The most obvious case is the 
simulator. A CRDR project could relocate, and in selected 
cases, extensively reconfigure a control panel subsequent 
to the simulator's design freeze. A mechanism to system= 
atically follow the design changes from the actual control 
panels through to the simulator is essential. Similarly, 
the design modifications could affect the content and/or 
sequence of particular procedures. As a minimum, CRDR 
projects tend to make extensive terminology changes 
which need to be incorporated into procedures. 

Other typical panel improvements involve modifica- 
tions which would have a direct affect on the content 
of training materials. A simple example is mimicking; 
the system line diagrams used in the classroom should 
correspond to the mimic arrangement on the panel as 
well as the actual system configuration. As for the 
procedures, terminology used in the training context 
should correspond to the operational language used on 
the control panels and in control room communications. 
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A mechanism which ensures overall operational/training 
consistency is necessary. 

Operating Procedures 

Operating procedures directly impact training curric- 
ulum materials and simulator training/ and are affected 
by the CRDR, The upgrading of plant procedures per 
CRDR operational te^rminology has been discussed. Further 
considerations regarding procedure interrelationships 
are the human factors aspects of the BPDS , the relation- 
ship between SPDS design/ procedure terminology/ and 
procedure restructuring to support the Symptom based 
approach* The resultant procedures provide the basis 
for "proceduraliEed" operational training/ most frequently 
conducted using the simulator. 

Training Curriculum Materials 

Training materials are on the 'receiving^ end of 
this process. The above discussed activities dictate 
changes to the curriculum. Training's role is to facilitate 
learning. To do this effectively/ plant configuration/ 
and procedure content and format are requisite inputs* 
The essence of the interrelationships is to ensure that 
training management and staff personnel are provided 
the necessary information to efficiently upgrade training 
materials . 

Other aspects of training which are integral to 
the overall issue of human performance/ are training 
feedback and training effectiveness* To close the loop 
on the overall network, feedback is necessary. This 
feedback relates to how well operators and teams performi 
and how well the training materials prepare operators 
and teams. 
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Simulator Training 
Simulators have gained broad industry acceptance 
and are being introduced into utility training programs. 
To this end, the thrust has been toward gaining the 
highest fidelity possible with the assumption that expoeure 
and practice is the road to improved human performance. 
From an Instructional Technology perspective, simulators 
are part of the training process/ but must be integrated 
with the overall training scheme. Therefore^ the training 
conducted using the simulator is in support of an opera^ 
tional training curriculum, and must be developed and 
conducted within that curriculum. These requirements 
dictate the need for a mechanism which ties the human 
performance activities together* 

THE INTEGRATION PROCESS 

In order to maximize the multi-disciplined team's 
efforts, a system is needed to efficiently track and 
integrate individual activities. The application of 
Personal Computer (PC) based CAD/DBM provides this capa-- 
bility - 

The Process 

The detailed process for establishing the CAD/DBM 
system is extensive and beyond the scope of this summary, 
A synopsis is provided which illustrates the functional 
steps involved in developing the DIM system and integrat- 
ing CAD with it. Appendix A to this paper provides 
additional insite into data-base relationships. 

Seven basic steps were involved in the design of 
the system: 
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1. Deheirmining and defining the need, 

2* Defining the Activities which address the need, 

3, Selection of Relational Database technology, 

4, Defining the relationships between Activities 
(including CAD ) * 

5, Defining specific databases which support the 
relationships . 

6 , Defining Outputs addressing user needs * 

7 , Implementa t ion/debugg ing system * 

Important characteristics of this system includes 

o All databases should be able to be accessed from 

any other database* 
o Design and authoring occur/ with the creation of 

the database. Subsequent manipulations occur 

within the DBM system, 
o CAD is to be a closely coupled subset of the 

DBM system, 

o Relationships designed to facilitate database 
output tailored to user population* 

THE POTENTIAL 

The potential of such an approach to industry has 
vast ramifications* This approach is applicable regard- 
less of work completed and/or planned/ in any of the four 
activities discussed. The most intriguing and beneficial 
aspect of this system is its ability to enhance and 
strengthen existing mechanisms. The logical extension is 
the "ultimate realization of optimized human performance* 
Database manipulation provides a constant monitoring and 
integration of activities which guarantees improved 
human performance * 

Once the system is functioning/ the expansion to 
address further needs is limited only by the creativity of 
those responsible for improving human performance* 
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APPENDIX A 
Functional & Relational Interaction 



Figure 1.0 illuatrataa the fundamental interaction o 
the DBM system, while Figure 2.0 illustrates the rela- 
tional interaction. The fundamental intteractiona are the 
Design/Operational activities of the CRDK and Procedures 
which act as the initiating activities. The Training 
Materials and Simulator activities are the recipients and 
facilitators of the Design/Operational activities. The 
CAD/DBM block is the common link. The Reporting Database 
provides for the systematic selection and retrieval of 
information/ for documentation . analysis, and reporting. 

The relational interaction (Figure 2.0) is an exten- 
sion of the fundamental interaction. This figure defines 
the database modules within a given activity, and illus- 
trates the relationships. It should be noted that the 
databases, can be accessed or manipulated on any field in 
any database. 



Pigure 1.0 



Figure 2.0 
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DEVELOPINS THE DESIRE FOR CHANGE 



Bin a. Gooch 



ABSTRACT 

This article leads the reader through the process of 
developing human potential by examining the assumption that 
development and change are synonymous concepts, and that 
many basic assumptions held regarding human reaction to 
change are false. If trainees proceed down the path filled 
with false premises^ their task of improving perFormance 1s 
made more difficult* 

After resolution of basic truths concerning the nature 
of change^ the article examines the process of creating a 
desire within individuals that will lead beyond motivation 
to activation. The article attempts to identify and discuss 
factors that are important to creating within Individuals 
the desire to Improve their performance. 



DEVELOPING HUMAN POTENTIAL 



Introduction 

The purpose of each article contained in this publication could be 
classified In one of two ways. The art1cle*s purpose Is either to make 
new things become familiar to the reader , or to present familiar things 
in new, interesting and, hopefully, useful ways* It should be obvious 
to the reader that this discourse on developing desire fits within the 
parameters of the second purpose. 



The training challenge is to develop human potential* In this 
developmental process^ trainers are concerned about four primary factors. 
These factors relate to increasing the knowledge hase of the trainee, 
developing new (or refining old) skills, shaping proper attitudes and 
developing appropriate habits. Professional development and improvement 
can result only from development within these four areas. 
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Developing KASH 

The problem faced in developing human potential within the nuclear 
training field is the same as the problem faced In improving the 
potential of the people of the world. The solution to improving 
performance is known by most but understood by few, i.e.. each person's 
KASH Formulal must be Increased. The formula (K+A+S+H= 
Improvement) appears simple; the process, however, is complex. 

It should be noted that there is a natural progression contained in 
the KASH Formula. The knowledge base that one possesses is instrumental 
in forming the attitudes that one has. An adequate knowledge base and 
an appropriate attitude are important in the development of desirable 
skills. And the application of desirable skills over time leads to 
desirable habits^-that Utopian state in which everyone takes the appro- 
priate action and uses the appropriate method (does right things right). 

Perhaps one of the reasons why development of the human resource is 
seldom given the attention which it deserves is that the process is so 
deceptively simple. Because every human being is different, each person 
responds to training differently. The uninitiated are likely to hold 
the belief that the important factor in the training process is to be 
firm, fair and friendly; and fair, they interpret to be treating 
everyone the same way. 

It is not the intent of this paper to make the point that the equal 
treatment of unequal s may very well be the highest form of inequality, 
nor is 1t the paper's purpose to point out that one of the greatest 
paradoxes within the training and development field is that frequently 
trainers accept the fact that everyone is different, but persist in 
treating everyone the same way in the name of fairness. The intent is 
to show that one of the trainer's major tasks is to create the desire 
for change within each trainee, and success in this developmental 
endeavor is directly related to the trainer's proficiency in recognizing 
individual differences, managing change and achieving activation. 



Managing Change 

A well known baseban player was quoted as having saids "It ain't 
over 'till it's over*'* This quote is included for two reasons. First, 
often there is value in stating the obvious, and second the author feels 
that the most appropriate way to introduce this section is by making the 
following obvious statement: Unless people changes they remain the 
same,* 

The primary factor to consider in the task associated with the 
training function is human reaction to the process of change* All 
training and development activity is directed toward changing the 
trainee. This change may relate to developing within an individual the 
ability to think differently, perform differently, or to feel 
differently* If it is given, then, that a basic truth within training 
is that developing human potential involves stimulating people to change 
in desired ways, then it is Important for trainers to have some under- 
standing of human reaction to change. This understanding is important 
if one is to succeed in developing within a person the desire to change* 

A good place to start in examining basic principles underlying the 
change process is with the commonly accepted truth, "People resist 
change**' If one were to conduct a survey to determine how many trainers 
agreed with this commonly held truth, it would probably be discovered 
that most do- Because developing human beings to perform in desired 
ways is so Interrelated with change, it is important to examine this 
"basic truth" more closely* For 1f the statement is indeed trues then 
the basic assumption would have to be made that people resist improving 
their performance or developing their potential. 



*By making two obvious statements, it was hoped that the author's 
anxiety level would be reduced by the fact that he was not the only one 
who may have wasted peoples' time by asking them to consider the 
obvious. 
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Thoughtful consideration of the statement, "People resist change," 
would likely lead individuals to change their opinion regarding what 
they believe to be true in this area. More defensible positions 
regarding this Issue would probably be (a) people resist some change and 
(b) people resist being changed. If people are convinced that the 
changes will lead to their improvement, they are less likely to resist 
the change. There is, however, a force within each of us that causes us 
to clTng to the familiar and resist the unknown. Whether you label this 
force maintaining equilibrium or homeostasis, the force's function seems 
to be to control excessive change within an individual. 

To illustrate how these two truths may work in an individual, 
consider the following illustration. You have just been informed by 
your supervisor that, because of your extremely high standard of 
performance, your salary will be doubled. Do you resist this change, or 
are you more likely to wonder why this increase had not occurred sooner? 
Even though you may actively welcome the change, you do not perceive 
that the change will cause you to change the way you are. You probably 
feel that, even though you will have twice the income, you will be the 
same person that you were before. 

The second factor that should be considered in the change process 
relates to a person's reaction to being changed. Most people perform 
the way they do because they have always performed that way before. 
Having performed in a particular manner over time, they have developed 
proficiency. Old habits become old friends. And Just as individuals 
feel a deep sense of loss associated with the death of an old friend, 
they feel a sense of loss at having to give up familiar ways of doing 
things. Even if one is assured that he/she will receive a better 
friend, the sense of loss is not totally negated. The point is that 
even though improving the person's performance is the objective of 
training, one should not be surprised to see negative attitudes during 
the change (development) process. 



III.BJ .5 



The title of this paper 1s "Developing the Desire for Change," 
Basic considerations concerning human reaction to the change process 
have been presented; however, a missing aspect at this point is careful 
consideration of this phenomenon called human desire. 

Creating Desire 

If one were to investigate the psychological and sociological 
literature associated with motivation a definition not too dissimilar 
with the following may be discovered: motivation is an internal 
construct, intervening between stimulus and responses which fashions the 
nature of human response. 

Perhaps one of the primary problems associated with human resource 
development is that those who are responsible for shaping human 
performance must understand motivation; whereas, those who write and 
share their research findings about motivation do so in language that is 
not easily understood. 

To demystify this force referred to as motivation, please consider 
the following simplified definitions. The first consists of a two-word 
d^'^^'^ltion that the author will call "want to". The second definition 
c Moists of three words-="desire for change". When it is said of a 
person that he/she does not seem to be properly motivated to take a 
desired course of action, all that is really being said is that the 
person has insufficient desire or does not want to. This statement 1s 
based on the assumption that the person ha? the competence to perform 
properly but seems to be lacking the will to perform properly. If the 
situation exists in which an individual lacks sufficient skill, then the 
problem is one of lack of competence rather than lack of motivation (or 
desire). 

Two of the major contributors to present-day practice in the area 
of motivation would seem to be Maslow^ and Audrey3, Most people who are 
associated with education^ training, development or management are 
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famm'ar with Maslow's theory regarding a hierarchy of human needs. 
This hierarchy, classifying human needs (and wants), starts with ba: 
physiological needs and moves through security needs, social needs, 
esteem needs ultimately to something that Maslow referred to as 
self=actual ization. 



Much of Audrey's work reveals that people strive to achieve 
security, stimulation, and identity and thereby eliminate feelings of 
anxiety related to insecurity, boredom, and anonymity. 

A study of motivational literature represented by the works of such 
people as Maslow and Audrey would lead to a few basic assumptions that 
may be important to .creating a desire for change within an individual, 
Some of these assumptions are: 

1. motivation is based in relative dissatisfaction, 

2. motivation is an internal force (a desire within), 

3. one cannot motivate, one merely stimulates others, 

4. the probability associated with one taking a particular course 
of action relates directly to the strength of that person's 
desire, and 

5. If people are satisfied with the way they are performing, one 
Should not expect them to perform differently. 

After careful consideration of these basic assumptions, it would 
seem that the training challenge of shaping desirable human performance 
would be to create sufficient dissatisfaction within individuals 
regarding their present performance compared to desired performance. In 
addressing this challenge, trainers may need to address the following 
two questions: 1) What are the greatest sources of human anxiety, and 
2) What process can be used to adjust peoples' attitudes to the point 
where they will take the desired action? 

The literature relating to human anxiety reveals that the three 
major sources of human anxiety and dissatisfaction are death, guilt, and 
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lack of purpose and/or Identity, It is probably no surprise to the 
readers that death and gui It made the 11st of human dissatisf lers, but 
some may be asking the question, Are death and guilt commonly used to 
create the dissatisfaction neCissi_ry to motivate someone to perform in a 
certain way?" 

One does not have to study Marine Corps basic training methods to 
realize that the answir to this flu est ion is, yes. Consideration of 
Maslow's hierarchy of human needs implies that death is the obvious 
result if people are deprived of nr eeting their basic physiological needs 
of oxygen, food, clothing and shel ^er. In addition, consider the 
possible guilt feelings associated with a person's inability to perform 
In desirable ways especially at th e esteem level. 

It is obvious that people who are responsible for development of 
the human resource need to m man v forces that are intended to create 
this desire for change. It is nec essary for this desire to develop to 
the point where individuals will ^ ant to perform differently. To create 
this desire^ it is necessary to ex amine the answer to the second 
question. 

There would seem to be four a^djustments in attitude necessary to 
create desire that is strongenoug h to cause Individuals to take proper 
action* The first step in this at ^Itude adjustment process is to make 
the individual aware of the dislra^ action or desired performance. 
Second^ the person should undirstamd the performance expected. Third, 
the individual should develop cone ^-ern regarding this inability to 
perform the desired task. And fourth, the individual should become 
sufficiently dissatisfied with h is/her presentepf ormance capacity when 
compared to the performance desi ^T^ed, Action results from the 
dissatisfaction associated with iieing unable to perform in the desired 
fashion. 

To summarize the purpose of this article, the following example is 
presented. Assume that Johni a ^ery valuable operator ^ was dangerously 
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ob^ss, and you wanted to stimulate him to lose weight, the desired 
action is weight loss on the operator's part. MaKing hiiaware of the 
pt-oblemis the first step 1o attitude adjustment. 

Mil] those who are awat-e of the problem take the action desired? 
Ptobably not. The next step is to make certain that he fully 
understands the problem. Information shared with the operator to ensure 
understanding will not likely cause him to change his Mavior regarding 
his. eating and exercising habits. 

The next step would be to create a concern within Hi operator 
m arding his present perfor-mance in relation to the deslrsd performance. 
One maywant to use death as the anxiety stream to cre^ieconcem within 
the opirator. Actuarial tables showing higher mortal i tj/ rates for obese 
peo^limay create concern wi-thin the operator. The world is fined with 
people who are concerned about their weight, but who do mt do anything 
abOL^t it, 

The operator must reach the last level of attitude adjystmerit to 
take appropriate action. He must be sufficiently dissatisfied with the 
way he is in relation to the way he should be. Many people are eating 
and exercising properly becai^se a doctor may have told thithat If they 
did not get their weight under- control, they would not be around very 
lonr - They must have suffici ent desire for the change, ad the doctor 
prov Tdedthe proper stimulation. 

Oolin's behavior changed through a very logical prOQmof attitude 
adjustment. The process invoTved awareness, understandlni, concern, 
dissatisfaction and behavior change. There is a tendency ii the 
train ing area to fail to cons t der these necessary attitude adjustments. 
Why something should be perfor-nred 1n a particular manrier my be Just as 
ifiipor -tant as how to perform pr~operly. 
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tJ^^y ^TRAI NING RELMCSP CMSES 
" PERSONNEL PE ^RFgRlgMCE 

PROBLEMS : 
CTrrB ROLE OF QRGANIgftCTN^L 
VALUE SYSTEM! 



ABSTRACT 



TraL^ning is fraquentiy viiwed as 
a panaGsL for all personnel related 
problems. whenever an ineK^lisable 
incident occurs, training isVirbaliEed 
by manv a_s the solution/ mnm^ 
as the da^use of the incident. All to 
frequentl_^^ training organtgations are 
needlessL_y Gommissioned to develop some 
elaborate course, only to flt.8Cover that the 
training is irrelevant to tbe problem. 

WhiL_e training as a sotntion 
to an incident is socially acceptable, its* 
use clQUd_a the real issue ^ Thi purpose of 
this pap^r is to emphasise that there 
are many reasons people do Wt 
perform, many of which are ^elated to 
organlEat^ional culture or vsluei. 
Values dL^xectly influenoe a^l 
activities which we are involved 
in, even to the eKtent of h^wwi 
view our^ selves and how we perceive 
our envi^^onment . They exert a 
strong* P-*ervasive influence on what 
and how w^e perform* 
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NON-TRAINING RF.T.ATED CAUSES O F 

PERSONNEL PERFORMANCE PROBLEt|g ; 

The Role of Organizational 
Value Systems 

Training is frequently viewed as a panacea for all 
personnel-related problems. Whenever an inexplicable 
incident occurs, training is verbalized by many the 
solution, or worse, as the cause of the incident^ kll to 
frequently, training organizations are needleSsi^ 
commissioned to develop some elaborate and expensive 
training course, only to discover that the tjralri^jins 
irrelevant to the problem. 

Various reasons account for this tendency tQ attach, 
training to personnel-related incidents. FoteinoScit is 
socially acceptable; that is, training is a " aaeg" solatioi 
for the personnel involved, especially for the indWclua.1 
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who is organizationally responsible and is being pressured 
regardi — ^ng the incident, its* cause and its* fix. It is 
easier" to simply say that the personnel involved did not 
knew did not know enough than to admit any of a_ host of 

other ^^ossible causes or worse, that they themselves 
had anY~ link to inducing the incident. In other words ^ 
the ina_ividual responsible for resolving the problem^ 
or theL_r superiors who are doing the pressuring, may 
be^ at least indirectly, a contributor to initiating 
the proiistolem, 

Mo^st people will agree that individuals who are 
didiGat_ed, caring and who are genuinely interested in 
ensuriix^g that everything is "right" are desirable 
employfe^es, whether they are operators, technicians, 
enginefe = rs, supervisors or managers. This person is one 
who is characterized as being self ^directed, naturally 
curious , competent, able to see problems and resolve them/ 



161 



IlIJ.2,4 

gets thing done even if not asked, and goe^ out of their 
way to be helpful to others. At the supervL^sory or 
managerial level, this individual has litti m difficulty 
communicating with others, instills feeling s of mutual 
respect and personal dignity, is confident ^-^nd 
self-assured without being arrogant, is trusting and 
honest with superiors and subordinates, enc^=urages 
openness and honesty in subordinates, recog^iizes the need 
for and uses appropriate managerial styles wwith differing 
people, knows where the organisation is heaSed and how to 
get involved. in other words, it can be sat^d that they 
are good managers, are altruistic, possess l_nsight and 
their orientation is both towards people anS. their 
accomplishments • 

While the connection between these typ^s of 
individuals and the frequency of errors is r^ot direct, it 
is certainly safe to say that the probabili^^y of gross 
errors involving dedicated, committed indivl duals is 
greatly reduced. 162 



So ttim paramount qt^estions are how do we instill 
or encout&g € these desirable traits and their inherent 
behaviors ciid how do we xnaintain them? Why does it 
seem that t liere are so ^ people in organisations^ 
especially large organisations^ that manifest these 
behaviors? To discover the answers to these questions, 
we rnust to some degree^ lemove ourselves from the trees 
so that oan see the :^orest; the forest being the 
envlroninent or organizational culture in which 
behavior t^3<es place an^ the individual interpersonal 
interactions which our &\ilture promotes. 

All S^liavior is a result of the stimuli provided by our 
environnien^ . How we respond to these stimuli are in part^ a 
product o£ our organizational values system which is the 
heart of oM^x culture* '^liese values, which frequently are 
not Verba IL Eed directly^ (which represents a value in 
itself), provide us witt^ what the organization views as 
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important^ good, worthwhile and their converse. They also 
help shape our perspectives in terms of how we view 
ourselves, the people around us and our jobs. 

To some small degree, organisational values are 
formulated by individual organiaational managers, 
especially when the organization is geographically 
isolated from th^ larger superstructure of the company. 
But in most other cases, the senior manager of the 
company sets the values. in either case, those values 
are filtered, massaged and individualized as they get 
passed on down through the corporate hierarchy, 
ultimately reflecting the personal beliefs and values 
of the respective individual. Moreover, senior 
managers tend to have value systems whj ^i^c: ^ ^;sistent 
with that of the larger corporation, o = : - not 
have been selected to move up the corporate hierachy. 
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since values are not directly communicated^ 
employees at all levels learn the values at-play 
through their perceptual filters and individual 
eKperiences, They see what types of things are 
emphasized, rewarded or frowned upon, whether they 
involve personal interactions, managerial styles, 
equipment, policies, communication itself, etc. 

If the organization emphasizes such values as 
open^Gommunications , personal well-being, trust, 
cooperativeness and interpersonal respect (at all levels), 
communicates these values and behaves consistently with 
them, then the associated behaviors have a high 
probability of occurring. If the organization 
emphasizes things or objects such as equipment, 
schedules, policies, etc, then our attitudes and focus 
will be directed toward these objects, perhaps at the 
expense of the human element. The fact that people are 
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involved in all these things, or are required to 
accomplish these objects, may become cursory or be 
taken for granted. Furthermore, the fact that people hav 
needs, and those needs must be satisfied for those 
desirable traits to manifest themselves, becomes 
unimportant, unattended to or ignored. Ultimately, goals 
equipment maintenance, safety, etc, becomes depersonalize 
and entities onto themselves. Problems related to these 
issues become safety problems, maintenance backlog 
problems and control problems. As Mr. Ackoff cogently 
said at a recent seminar^' "there is no such thing as a 
"safety" problem or a "backlog" problem but rather these 
labels are attached to symptoms which represent the 
perspective of the individual who is doing the labeling". 
We have become so object oriented and specialized that 
when we analyze things that are not what they should be, 

1* R.A. Ackoff, Keynote Address, Con Edison CGntral 
Operations Seminar, January 1987 
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all we can do is relate to them from our own narrow frame 
of reference. We then put labels on them, such as valve 
line-up problems, engineering problems, training 
problems, etc. Our labeling tends to encourage a more 
narrow definition and further removes us from the ability to 
see the big picture, as well as give us a false sense of 
security. We sometimes have difficulty seeing that the 
tardiness problem, maintenance backlog problem and the plant 
trip problem may all be related and symptomatic of one big 
problem; our fundamental inability to see the 
interconnection of things and people and to provide them 
with an environment which is conducive to satisfying their 
needs and that clearly encourages those traits we all value* 

The most dedicated, competent and committed 
individual can become demoralised, demotivated, and 
alienated given an environment which is inconsistent. 
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ambiguous, s browbeats people or doesn't provide 
avenues of communication, rewards the wrong people and 
goes out of their way to control every asrect of one's 
behavior . 

Companies must recognize that over time and with 
differing g-nerations of employees, the value systems of 
individuals vary. No longer is security a primary value 
younger employees (not that security isn't important). 
These employees want and need to be involved in decisions 
affecting them, they want and need to receive feelings of 
accomplishment and well-being from the work they perform, 
they want and need an environment that will allow them to 
express their thoughts and feelings and above all, people 
want and need to feel good about themselves, by giving it 
their all and by contributing successfully. Therefore, 
it becomes clear, when organizations investigate deeper 
than the surface to understand many performance problems, 
that the culture, the value system of the organisation, 
must be looked at in terms of the behaviors it conditions. 
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The following paper was not received in time to be published in the pro^ 
ceedings. Space is provided below for notes. 



Mental Models for Expert Systema to Technology of Training.*.*.*.,,*.., 
Dr. Harold S. Blackman, EG&Gi Idaho j Incorporated 



NOTES 2 
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mY TRAINING DOESN'T TAKE 



T. R. Hyldahl 



ABSTRACT 



For soine reason^ In spite of effective program design and devel- 
opment and excellent instruction, trainees don't perform as expected 
when they get back to the Job, The Industry's initial response to 
this lack of performance Is to assume that something is wrong with 
the training. Our strategy has usually been to attempt to change the 
program or the means of instruction. What we fail to realize Is that 
trainees^ for some reason^ choose not to learn or to forget what was 
learned * 

This presentation will deal with a number of other factors that 
would account for trainee inability to assimilate or retain informa- 
tion. It will also include the strategies for determining what these 
factors are* ^ 

The factors that will be dealt with will include i 

1, Lack of attention 

2, Insufficient trainee knowledge (ineffective processing of 
new data) 

3* Emotional blocking of reception or memory 
4* No immediate :ipplication of learning 

5, Poor motivation for learning or remembering 

6. Lack of response to trainee needs ^ concerns $ or questions 



Each of uSj at one time or another , has fantasized being a famous 
detective. We are called upon to solve an impossible case, and, as a 
result of our brilliant analysis, clever strategy, and a dash of courage 
the mystery is solved. Today, we are all going to have an opportunity 
to be detectives and deal with the Great Training Caper* 

Here are the clues * There is a well designed training course. 
Not only is it well designed, it is also well developed. In addition 
it has been superbly delivered. The mystery iSs that in spite of all 
this preparation and skill, the trainee goes back to the job and does 
not perform as was expected , Scotland Yards the NRCj or your manage-- 
ments (whoever the foil for your intelligence is going to be) speculates 
that the solution of the mystery is to improve either the design^ 
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development, or implementation of the program. Over your objections 
they call In the designers, developers, and implementers and tell them 
to do a better Job. The designers, developers and Implementers try 
three times as hard. They even create multi=raedia eventa and have 
Robert Redford do the instructing. 

Still the trainees do not perform as they should. Scotland Yard, 
the NRG, or your management, throw up their arms in despair. "iJho can 
solve this problem?" they wail. This is your big opportunity, but 
where do you begin? 

You're probably saying by now, "This isn't that much ,of a fantasy. 
I've seen this mystery repeated over and over again in many different 
ways, and always with the same results." 

I'll have to admit, it has always been a mystery to me why so 
many individuals schooled in the scientific method forget all their 
training when it comes to the area of training. It seems like 
technical people say to themselves. "I'll go as far as I can systemat= 
ically, and then I'll depend on magic or guesswork." For the next few 
minutes I'm going to try and remind you about those skills that we have 
all been taug, , and how they relate to the solving of the training 
mystery. 

Let's attack our problem with the use of cause and effect. In 
this case the effect is that the employee is not performing as he/she 
should be. Now, as good scientists, we realize that it doesn't make 
good sense to Just guess at what the cause for that effect is. If 
someone were to tell us that the cause for cancer was radiation we 
would be offended by their lack of scientific sophistication in making 
such a judgment. But when someone says that a person does not perform 
on the Job as they should, and someone posits that the cause is training, 
even bright people are willing to accept that conclusion as reasonable. 

So what does the good scientist, or detective, do? First of all 
he would probably have more than one hypothesis as to what the cause 
for the effect might be. Take the person not performing on thft Job. 
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There could be any number of causes to account for that effect. The 
person may not understand whet has to be done^ he may not want to do 
what has to be dones he may not be able to do what has to be dones his 
supervisor might have given poor direction^ the procedures may be poor^ 
the equipment he has to work with might be poor, he may have forgotten 
how to do what has to be done, he may be on drugs, he may have had an 
argument with his wife at breakfasts or he's just too fatigued to do it* 

After having posited any number of causes to account for the 
effect, the second step that the good scientist or detective does is 
to systematically eliminate the possible hypotheses for the cause. 
This is a difficult step and one for which you will not find a whole 
lot of support. One reason, is that trainees and supervisors are some- 
what reluctant to be included among the variables to explain why training 
is not successful. It is far easier for them to explain the problem 
away under the definition of "training". As a good detective you cannot 
be deterred by the defenses of your suspects. Truth is always your goal, 
so you will unrelentingly pursue your hypotheses. 

There are any number of ways of going about doing an investigation. 
An academic researcher has an arsenal of tests and statistical tools, I 
perfer to ask the trainee. 

Some may perceive that asking the trainee is somewhat simplistic and 
not nearly as sophisticated as the other diagnostic tools at the disposal 
of the detective. My personal experience has been that the trainee is 
probably the best source of determining why he/she has not performed as 
was expected. This is not to say that this source is always reliable or 
even accurate. There are many times when the trainee source does not 
benefit from even being honests but then if a detective cannot distin- 
guish between honesty and dishonesty he is not deserving of the title , 
detective. 

To aid in the pursuit of the investigation, let me suggest some 
variables that you may consider when probing the trainee for possible 
causes of non-performance. 
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Pre-Tral nlna Knowledge I think we often forget that adults 
have to know certain things in order for the new knowledge to make 
sense. An absurd example would be to have someone explain something 
to you in Chinese. The explanation would be meaningless. Studies 
have shown that people retain a greater percentage of new information 
when they are already familiar with 40% of the information being 
presented. 

What many trainers, and managers of trainers^ forget is that 
people have difficulty assimilating new knowledge unless they have a 
structure or internal organization to deal with the new knowledge. If 
the parson has no place to store the information, nothing is going to 
happen to it. If I tell you that all gwumps are smerthed it doesn't 
mean anything unless you know what a gwump is and what the verb smerth 
means » 

One of our problems is the very short memory span of "short term 
memory". Short term memory only lasts for 30 seconds. Therefore the 
brain has to do something with the new information it is handling with- 
in 30 seconds or it is lost. What the brain tries very hard to do, is 
find a file in long term memory to put the new information into so that 
it can be called up again when it is needed. 

If the learner is having to build long term memory files at the 
same time he is processing new information, the circuits are over- 
taxed and the brain starts thinking of something more comfortable like 
food or other pleasures. 

From the detectives standpoint, one of the reasons that the train- 
ing is not taking may be because the trainee does not have enough "ad- 
vanced organizers" to handle the new information coming in. As a result 
no matter how good the material is, it still will not be processed. No 
training has taken place* 

How do you solve this problem? You can estimate on the basis of 
the trainees eKperience what kind of organizers he is likely to have or 
you can always ask him in different ways if he is following the discus- 
sion* 
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A good question to addrass for this discussion might be, '"What is 
an advanced organizer?" If you don't understand the concept, you 
may have difficulty following the discussion. 

#2, Lack of Attention Closely tied to trainee knowledge is 
the oft discussed area of sttention. We're all eKperts on how to 
get and keep, attention. We all have a long series of successes with 
spouses, children and trainees^ right? What most of us fail to 
remember when we are sharing all out wisdom^ is what we are competing 
with* Take this situation for example* How many stimuli do you have 
that are cQmpetlng with my talking to you? There are assorted noises , 
visual distractions, and any number of thoughts that you can call up 
that might be more interesting than what I am saying. You can also 
listen about saven times as fast as I can talk, so I am always running 
behind your thought processes. Unfortunately if you're like most 
adults it will be vary hard to keep your attention for more than eight 
minutes 5 so I've already lost most of you by now. 

Why any trainer is arrogant enough to believe that someone would 
listen to him for two hours is amazing to me. That does not mean that 
people, (especially trainees) are not capable of feigning attention* 
One of our early social skills is pretending that we are paying attention. 
It is considered polite to pretend you are listening. The only problems 
with this pretense is that it gives poor feedback to the trainer. This 
poor soul thinks that you are hanging on his every word and are inter- 
ested in what is being said* He also believes that you understand the 
material. It isn't until there is a test or you get back on the job 
that he realizes that you were faking attention* 

From the detectives standpoint ^ one of the reasons that training 
may not be taking is because the trainee is not attending. All the 
good information that you have to share is not getting processed be- 
cause the trainee is attending to something else* There are millions 
of pages written on how to gat and keep peoples attention* I won't 
take the time to reiterate all these ideas. What I have found over 
the years is that the only thing that most people attend to consis- 
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tently is themselvea. The best approach for maintaining attention Is 
assuring that the trainee is as actively involved as he can be. Only 
you know for sure how to do that in your area of expertise. 

Emotional Blocking I'm sure it comes as no surprise to any- 
one to share that your emotional state does in fact influence what you 
learn. Trainees that are angry, anxious, or depressed are not partl=- 
calarly receptive to new ideas or information. 

The anxious trainees because they are too busy dealing with what 
they are anxious about or their anxiety symptoms. If your pulse is 
147 you may not be thinking about valve maintenance. 

The angry trainee because the angry person is either focused on 
what he la angry about or, as Is often the case, is angry about having 
to be in training. 

The depressed trainees because they usually are not motivated 
sufficiently to feel that there Is anything that you might say or do 
that can be of any help to them. 

When you talk to people who are doing a lot of emotional blocking 
you receive "feeling" responses to the course. 

The anxious trainee will aay "The course seemed awfully long or 
confused." What they are saying is they were uncomfortable in the 
situation or because of their high anxiety everything seems confused 
to them. 

The angry person will almost always tell you that the course was a 
waste of time. Either he already knew the material, the material wasn't 
relevant, or he knew how to present it better. The bottom line is that 
the angry Individual usually feels undervalued, and he desires to be 
taken more seriously. The dynamics of how to do this is beyond this 
presentation, but do pay attention to those cues. 

The depressed person will usually tell you the course wasn't 
relevant or that it was boring. What they are saying is that nothing 
has any value and what you are presenting isn't any different. 
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So how does the detective determina if emotional blocking is the 
reason that the training is not taking? As has been hinted at, you 
listen to the amount of emotional raspouses to the contant. If your 
trainees are talking about ideas and content, the likelihood is that 
they are not doing a whole lot of blocking. If they are giving you 
all sorts of "feelings'' about the courses it is probable that some- 
thing is getting in the way of receiving your information. Of course, 
againj the way to find out this information is to listen to the students 
or ask them questions about the course* 

#4* No InuTiedlate Application People within a technical environ- 
ment always perceive of themselves as being "practical", "common- sense" 
types of people. This may be the case in other areas of their lives or 
works but it doesn-t seem reflected when it comes to training. Where 
else in the world would you tell someone how to do something and ask 
him to do it six months later? 

One of the most critical reasons that training doesn't take is 
because trainees don't have an opportunity to practice what they learn 
quickly enough. If people realize this, why does management wait so 
long after training to implement the new knowledge? My hunch is that 
people have a false assumption about how the human brain works. For 
years I have been hearing strange statements like, "We only use 20% 
of our brains". I have never figured out what that means, but it 
does lead people to visualize the brain as a bottomless pit into which 
you can continually drop information and retrieve it whenever desired. 
From our own experience , we know better, 

I think we would be far better off if we compared the brain to a 
computer, and thought of storing information in terms of an elaborate 
filing system. This will help us to answer some very practical questions 
about information. 

What happens when we receive a letter for which we have no file? 
We either create a file or we throw the letter away. The brain does 
the same thing. 

What happens when we run out of room in our filing cabinet? Unless 
we like to proliferate filing cabinets all over our world, we clean the 
cabinets out. We take the items we haven't been using and we dump them. 
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The harsh reality is that we do the same thing with our brain. Now 
this isn't as conscious a decision as cleaning our desk drawers and 
we certainly don't get a printout. What we do get is, "I once knew 
how to do this calculus problem but now I can't do it." 

Another example that is closer to computer filing is when nhe 
information la still there but we don't remember where it la filed 
or what commanda to use to get to it. 

Mho was the singer that sang "Mule Train"? Many of us, no matter 
how young, may have that Information stored someplace. The question la 
how do you get to that knowledge? My way was by seeing a picture of a 
wavy haired guy with a mule train on a TV show. Your way may be some 
completely different approach. 

The bottom line on this discussion is that the brain may not be 
able to retrieve information if it does not use it for a while. The 
brain may have dropped it from it's filing system or it may not recall 
the commands necessary to get to it. 

The good detective when confronted with a trainee that doesn't 
perform as he should can always ask when the desired information was 
learned, how quickly it was practiced, and how recently it was practiced, 
Having this Information, you can allow "common sense" to be your guide. 

#5. Training not Responsive to Trainee Needs There is something 
very self-centered about the average adult. For the most part, he 
desires something that la beneficial to him. For some unknown reason 
he finds It difficult to retain information for which he can find no 
useful application. The average adult will usually seek out the types 
of things he needs to know. If he has to fill out a tax form or change 
spark plugs, he will find out how to do it. 

What the average adult has tremendous resistance to, is someone 
else telling him what should be Important to him. All you have to do 
to rile up your neighbor or peer is tell him what he should learn for 
his own good. Even worse Is to demand that he learn something for 
which he perceives he has no use. 
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Probably one of the most useless things a trainer can do ±s to provide 
an answer to a question that no one has asked* Ideally training should 
be based on verbalized questions of the trainee. More realistically, 
the training can be based on the anticipated questions of the trainee* 
Where we often mess up is by answering the questions that we as trainers 
are interested ins and assume that the trainee is interested in the same 
question. The best means for determining what the trainee may be inter^ 
ested in. Is some form of survey. Probably ^ the more informal the survey 
the better* 

From the detectives standpoint , there is a good possibility that 
the reason that the training didn't take is because the trainee didn't 
perceive that there was any need for it in the first place. One of the 
difficulties the trainer has is that the trainee usually doesn't share 
this information with the trainer. The average trainee allows th^ 
trainer to go on teaching the irrelevant material without complaint. 
After all, one wasn't allowed to ask for relevance as a child, and 
many adults fail to realiEe that they have the right to ask for it as 
an adult* 

Your best bet as a detective/ trainer is to create a climate where 
the trainees share their feelings about the content of the training. 
It will help you to better determine needs and also to find out why 
certain information isn't passing the test back on the job. 

We're coming to the stage where we're close to overloading your 
receptors, but there is one more area that shou^^ be mentioned, its the 
somewhat cliche oriented area of , * , * 

#6* Motivation There has been a lot of time spent on trying to 
figure out what motivates someone else. The library is filled with 
books on the subject* There are movies on it, and you could probably 
have a vendor at your office by Monday to present a program for all 
your employees. The unfortunate things is, no one is quite sure what 
motivation means or how to do it. This is another one of those 
"common sense" areas. 
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Oftentimes when someone doesn't perform as we expect him to, we 
say he isn't motivated. The term "motivation" becomes a catchall for 
all sorts of things we don't know how to explain. I have a hunch that 
from a training standpoint we're better off without the term. 

My reasoning is as follows: IJho can motivate the trainee? Ulti- 
mately, only the trainee, right? Who is responsible for the trainee 
learning? Again the trainee, right? Can you make someone learn that 
doesn't want to? Of course not? What is your responsibility to some- 
one that doesn't want to learn? None... So the concept of motivation 
is only of some value if you feel you have some responsibility to have 
someone do something that thay do not choose to do. In other words, to 
go contrary to their will. There are people in the world that may have 
those responsibilities. I find it hard to believe the trainer is one 
of them. 

What is helpful for the trainer is the realization that there Is 
a cause for every behavior. People do not wander around behaving random- 
ly. They really are trying to achieve something by behaving the way they 
do. The detective/trainer has to figure out what that is. 

How does this apply to training? Returning to our initial problem. 
Why would an individual who has been taught to perform a certain function 
not perform it well? There could be a multitude of reasons. Why don't 
I wash dishes well? It has nothing to do with ability or intelligence. 
I don't want to wash dishes, and I found out as a youth if I did it poorly 
enough, no one would ask me to do it again. Not very noble, but ef- 
fective. 

A trainee does not respond to a question in class. Is it that 
he does not know the answer? Or la it he Is shy and doesn't like being 
asked questions? Perhaps he Is angry about being in class and la juat 
rebelling. There are any number of cauaes to explain a behavior and 
the detectives concern Is to determine which cause is the most plausible. 
Having determined the cause he may be able to do something about elim- 
inating the problem. 
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I place a big emphasis on the word ''may". One of the weak areas 
I've noted In our Industry Is the tendency to assume that we have solved 
a problem once it has been identified. Unlike television and novels 5 
real life does not allow us the luxury of basking in success once we 've 
eKplained our flawless analysis of the problem* The real world requires 
that others concur with our analysis, and that someone is willing to 
work to rectify the problem* 

Being a detective is not easy work* The reason that something 
Is a mystery is because the average person cannot solve it. The reason 
training does not take is not always easy to discover , I think that 
we're getting sophisticated enough to move beyond the "butler or train- 
ing did it". There are a lot of suspects; knowledge, attention, 
emotional blocking, no immediate application, not responsive to trainee 
needs* and motivation are just the most likely of the lot* 

Your most likely witness is the trainee himself* No effort 
should be spared in examining him to find out what he thought, heard * 
saw, and felt* If you remember what's Involved in a good diagnosis, 
you should be able to be a good detective* To quote Sherlock Holmes, 
who spent most of his time training a scientist, "It's elementary, my 
dear Watson", 
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SITUATIONS. THAT CAN HELP YOU FAIL TM 



NRq EXAMINATION 
J.I. MGMILLEN 

^STRACT 

Conditions whioh can lead to failure of the NRG exMlnation 
of parsons who have shown great promise during the training program 
are explorad. Savaral of these conditions bxb cited through the 
personel expariances of the author over the many yaws of adminis- 
tration of examinations. Although these things do not happen very 
ofteni It Is conaluded that training personnel shoiild be awM^e of 
what might happen and build on their relationship with the 
trainees so that thay can counsel then on the possible alterna- 
tives that thay have In the face of unusual circumstances. 
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GENERIC MODELINGS ENHANCING THE PRODUCTIVITY OF 



TRADITIONAL ISP METHODS AND PRACTICES 



Gary Zwissler 
Patrick E, Smith 



ABSTRACT 



Traditional methods of Instructional Systems Development 
(ISD) typically rely on task analysis for the identification 
of knowledge and sill requirements for successful task 
performance* Past experience with these methods has shown 
them to be quite labor intensive and costly. 

Since the results of task analysis are essential to writing 
learning objectives^ defining program content and preparing 
job performance measures, project managers traditionally have 
had two options for dealing with this requirement i 1) 
Conduct Task Analysis - schedule the time and commit the 
resources to complete this activity in the traditional 
manner, or 2) Omit Task Analysis - develop instructional 
materials without the requisite core information provided by 
the analysis. 

in response to the need for a task analysis In support of 
systems training for plant operators at San Onofre Nuclear 
Generating Station (SONGS) , an alternative task analysis 
process was developed to meet the constraints of limited 
project time and resources* The process, based on generic 
modeling, identifies task analysis data which applies across 
groupings of tasks for most plant systems. 

Generic models represent the knowledge and skills required 
for the operation of any plant system. These knowledge and 
skills are comprehensive and are derived from an analysis of 
the operating requirements of a typical plant system. When 
applied to the task analysis process, these system operating 
requirements are equivalent to groups of tasks logically 
organized by their commonalities. Attributes, common to all 
tasks within each group, provide the essential operating 
requirements for each system. A generic model is created by 
validating and sequencing these similarities into an order 
which represents the training needed by the operator to 
successfully perform the tasks associated with the syetem for 
which training is being prepared. 
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SJi^S ^ t"^ generic models requires careful analysis for the 
common elements of tasks across systems and a careful 

°^ "^^^^ P«rformanL reguiremSnls 
Hypothetical models must be validated with the job incumbents 
and revised to achieve the greatest possible fidelity bSlweIn 
tSj jS? ^"^ performance objective and the reguirementrol 

Applying generic models as part of the traditional ISD 
ESu;rtir°^''^? developing and implementing methods which 
ensure the consistent application of the model during the 
^^J^^^^^^^^P^^^ Phases. Many of these meSodI beLme 
the basis for improving the management of training 
development projects and provide the basis for maintaining 
the training system in the future. in^aining 

^^Ji?! models provides both quantitative and 

qualitative benefits to development projects. For those 
loni??^' must be completed with" resource and time 

constraints, generic modeling offers a systematic approach to 
conducting task analysis. Instructors ulinrmateriair 
developed with the aid of generic models re|or? griJter 
consistency among the instructional components than 

Sol^V°"^-L^^^^i*^^^ materials developed withSut generic 

The net effect is increased utility of lesson^ 
?Seir us:%'„'^^^''^? applications for generic models include 
Jnsirualioial^Itriti^JIs!"" identifying 
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INTRODUCTION 



This section describes the relationship between classical 
task analysis methods and those of generic modeling. Both 
approaches are reviewed within the context of the 
performance-base training evolution currently taking place 
within the nuclear industry. The following topics are 
addressed; 

m traditional task analysis methods^ 

m an alternative task analysis method. 



Traditional Task Analysis Methods 



Background The role of task analysis in instructional 

systems development is regarded by most 
program developers as an essential step in the 
process. Some of the reasons Include: 

m the identification of critical informa- 
tion needed by the worker to competently 
and safely perform his job^ 

m reducing the possibility that essential 

knowledge and skills will be omitted from 
the training, 

• providing a basis for the remaining steps 
of the program development process . 



Constraints Regardless of the potential impact that task 
of Task analysis can have on training program 

Analysis development, the decision to engage in this 

activity is not always an easy choice for 
training management. There are typically two 
reasons for this; they are as follows: 

• Resources; task analysis is very labor- 
intensive; it can consume 15 to 35 % of 
pro j ect resources , 

• Time; on average, 12 tasks per analyst 
per day , 



IV,A.K4 

Traditional Task Analysis Methods, continued 



While resources and time are two important 
variables that are often considered in the 
decision to conduct task analysis^ there is 
also a third factor that management usually 
must face I that is, 

"What will I gain from engaging in this 
activity?" 

When the anticipated results of the process 
are described to management, the usual 
response is often, 

"Why do we need to analyze something we 
already know how to do in order to 
document information that is currently 
available in existing training material 
or plant procedures?" 



Concerns of 
Management 



Classical Classical task analysis proceduras require the 

Task ^ Training Analyst (TA) to focus on what the 

Analysis worker does under the conditions that the job 

is normally performed. Typically, the 
procedure requires that the TA begin byi 

• reviewing all the major tasks for the 
job, 

m listing the subtasks and steps included 
in each task, 

m identifying the knowledge and skill 

requirements for performing the task; 
i,e,, "What is it that a person must know 
or be able to do in order to complete the 
specified activity? 

When the TA is satisfied that all that can be 
known about the performance requirements of a 
given task have been documented, another task 
is selected and the process is repeated. 
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Traditional Task Analysis Methods^ continued 



Classical Analysis continues in a straight ^line fashion 

Task until all tasks that have been selected for 

Analysis^ training have been analyzed. This process is 

continued concluded with the writing of learning 

objectives which are derived from the 
knowledge and skills identified for each task 
or task element. 



Instructor The increased emphasis that the nuclear 

Impact industry has placed on the benefits of task 

analysis as a basic tenet of performance- 
based training is well founded in the 
"value added" to the quality of training 
programs and materials* However^ the 
requirement to engage in this process adds 
additional complexity to the development of 
training material faced by the average 
instructor, for example i 

• pressure from working in a high-risk 
training environment , 

m new development processes which are a 
significant departure from past 
practices, 

• a shift from content to performance 
objectives as the specification of what 
should be trained, 

• a lack of training and experience using 
task analysis and ob j ectives development 
methods . 
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An Alternative Task Analysis Method 



Improving Task analysis as it is conducted in the real 

^^^^ . world with GompleK systems and unavoidable 

Analysis constraints, usuallv departs from straight- 

Methods line or classical methods. Often the approach 

taken by the analyst working under imposed 
tlme^resource constraints amounts to little 
more than using rules of thumb or educated 
guessing. 



roving Clearly there is a need for a process that 

^^^^ combines the rigor of classical task analysis 

Analysis methods with the availability of, and 

Methods, orientation toward instructional content 

continued This should result in an accelerated and less 

costly approach to the task analysis and the 
development of learning objectives; one which 
is compatible with the systematic development 
of performance-based training programs. 



A system of generic models for conducting task 
analysis and writing learning objectives, such 
as those prepared at San Onofre for the 
development of Operator training programs 
seems to offer an effective alternative to 
traditional task analysis methods, 

A key factor in the developing generic models 
IS information; it is the primary outcome of 
the task analysis proc^oss. Emphasis is placed 
on what is already known about what a worker 
needs to know or be able to do in order to 
perform certain types of tasks. 

Essentially, generic model methodology 
maximizes the use of the information that is 
available at the time the requirement for 
performing task analysis arises, it takes 
advantage of the fact, that for a given unit 
of work related tasks, such as those 
associated with the operation of power plant 
systems, general categories of information can 
be applied to a specific tasks within that 
work unit. 
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DEVELOPMENT AND IMPLEMENTATION OF GENERIC MODELS 

This section describes the methods for developing and 
implementing generic modals. The following topics are 
addressed I 

• An Introduction to Generic Modeling^ 

• The Generic Modeling Process Flowpath, 

• Descriptions of the Process Steps ^ 

• Process Control Methods, 

An Introduction to Generic Modeling 



Background GeneriG modeling is a method of conducting 

task analysis which analyzes groups of tasks 
for their common attributes^ then generalizes 
the attributes to a task family . 

Definition I A task family is a group of tasks 
which share a set of common characteristics. 



When to Use The most appropriate use of generic modeling 
Generic occurs in large scale task analysis efforts 

Modeling where; 

• a high ratio of tasks to task analysts 
exists^ 

• the tasks may be grouped into families, 

• the time and resources available for 
traditional task analysis is constrained. 



188 



IV.AJ .8 



An Introduction to Generic Modeling 
Continued^ 



Benefits When used in a task analysis project^ generic 
modeling provides the following benefits: 

• a viable task analysis approach for projects 
with resource and/or schedule constraints, 
and, 

• a controlled process for developing lesson 
objectives. 



Requirements The use of generic modeling requires strict 

adherence to the following requirements in 
order to ensure the quality of its results; 

• development of a strictly defined 
process. 



documentation of all activities, 

defined methods for project management, 
and process control, 

aommitment to ongoing maintenance and 
support of the process* 
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The Generic Modeling Process Flowpath 



Introduction The generic jnodellng process follows a series 

of well defined steps. 

Since all projects have their own imique set 
of constraints and problems , completion of 
each step may vary from one project to the 
next. Deviation from the methods described in 
this paper are permissible as long as the 
following criteria are met. 

Each step should be performed with; 

• a clear understanding of necjessary inputs 
and required outputs, 

• proper process controls implemented to 
ensure its consistent compietlon by all 
individuals involved, 

l,e,: Use of guidelines, checklists, 
review and approvals, etc. 

• appropriate documentation to allow for 
the repeatability and/or analysis of the 
methods used . 



Flowpath The flowpath diagram for the development and 
Diagram and implementation of generic models (stte Figure 1) 

provides a sequential listing of the steps to be 

followed* 

In the diagram, specific steps will be highlighted 
by capitalized text* These steps will be defined 
in the section. Descriptions of the Process Steps* 
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The Generic Modeling Process Flowpath 




Yes y 




DEVELOP 
KNOWLEDGE 
MODEL 








1 


4A 












Validate know- 
ledge MODEL 
W/EME ' m 




Add/Modify 
Knowledge 
Elements 


ALL TOPIC 


*V^Nd 







SMENTS 
PRESENT? 



Yes 



CONVERT TOPICS 
TO KNQWEDQE 
AND SKILL 
STATEMENTS 



VALIDATE MODEL 
W/JOi 
INCUMBENTS 



7A 



Add/Modify 
Knowledge 
Statements 



Figure 1, Generic Modeling Process Flowpe^th 
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The Generic Modeling Process Flowpath 
Continued, 




DEVELOP GENERIC 
OBJECTIVES FOR 
TRAINING 
PROGRAM (S) 



Author 
Reaeives LeesDn 
Authoring 
Asaignment 



10 



WRITE SYSTEM/ 
LieSON 
SPECIFIC 
OBJECTIVES 



Add/Modify 
Objeotive 
Statemente 




Figure 1, Ganeric Modeling Process Flowpath 



192 

o 

ERIC 



IV,A.l .12 

Description of the Process Steps 
Step 1, Projeat Meets Generic Model Criteria 



introciuction The first step requires that the project be 

evaluated for its "fit" to the generic model 
process , 



Judging the The extent to which a project lends itself to 
Fit generic modeling can be determined by 

comparing project conditions to a defined set 

of acceptance criteria* 



Acceptance Judge the applicability of generic modeling 

Criteria for a project by using the criteria from the 

table below. 



If . . . 


then , , . 


the project includes the following 
conditions: 

o a ratio of tasks to task analysis 
which exceed approximately 
100: 1, and, 

o the tasks can be divided into 
groups which share common 
traits, and, 

o there are limited project 
resources and/or time 
constraints. 


consider 
using the 
generic 
modeling 
process . 


the project includfes the following 
conditions: 

o a requirement that extensive task 
specific performance data be 
collected, and/or, 

o project resources and schedules 
support traditional methods. 


consider 
using other 
Task 

Analysis 
Methods . 
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Description of the Process Steps 



Step 1^ Project Meets Generic Model Criteria^ continued 



Example At San Onofre Nuclear Generating Station (SONGS) , 

developing the Operations Training Program required 
the analysis of all tasks assocJ Hi^ced with the 
operation of plant systems. 

The project conditions were as follows i 

• analysis of 2448 tasks, 

• three task analysts for a task/analyst ratio 
of 816:1, 

• completion of the project within 4 weeks* 

Note I Many things must be considered when 
determining scheduling impact. For this 
project, subject matter experts would only be 
available on a part time basis. As a result 
the projected daily throughput using 
traditional methods was approKimately 4 
tasks/analyst/day, yielding a total time 
required of 735 man days or 147 man weeks. 

Based on the project aonditions, a generic modeling 
approaQh was determined to be appropriate. 



Non Development of the SONGS Warehouse Training Program 

Example the analysis of warehouse operations task data. 

The project conditions were as follows i 

• analysis of 110 tasks, 

• a -^.ask/analyst ratio of 110 il, 

• completion of the project within 1 week. 

Note I The task analyst was an experienced SME 
available on a full time basis. Using 
traditional analysis methods^ the projected 
throughput was 3 2 tasks/day, resulting in a 
total time of 4 man days. 
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Description of the Process Steps 
Step 2, Develop the Knowleclge Model 



Introduction In step 2^ the foundation for the generic 

model Is created. The Knowledge Model forms 
the basis from which all other steps evolve. 

The Knowledge Model is developed by completing 
the following steps; 

1* identify the task families^ 

2* create the Generic Knowledge Outline for 
the task family. 



Task families are groups of tasks which share a set 
of common characteristics. Task families are 
usually based on groups of tasks which; 

• combine to form an expected outcome, and^ 

• involve similar equipment or processes. 

With these characteristics in mind, task families 
are usually identified as the Svstam/Dutv level of 
the job analysis hierarchy^ as defined in INPO 
document, 83-003, Job and Task Analysis Ugey^ 
Manual s 



Task 
Families 



Example The task families for systems training in the SONGS 
Operations training programs were based on the 
systems within the plant i 

• feedwater system, 

• component cooling water system, 

• 6,9 KV system, etc. 
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Description of the Process Steps 



Step 2^ Develop the Knowledge Model, continued 



GeneriG The Gsnerie Knowledge Outline is a 

Knowledge the Knowledge that is shared by the .ai^^k 

Outline families. The Generio Knowledge O^at iine 

developed by completing the follow ^ ' 



Step 


Action _J 


1. 


Determine the knowledge which are 
common to all task families. 


2. 


Order the common knowledge into an 
outline using a suitable sequencing 
strategy (see Figure £ ) . 

Note: Sequencing strategies are best 
selected according to the type of 
knowledge identified. The strategies 
most often used are: 

o logical order, 

o simple to complex, 

o most critical f irst . 

Example 1: In the SONGS systems lesson 
plan project, knowledge v/ere sequenced 
based on the logical order implied 
by the equipment being operated. 

Example 2: Knowledge dealing with the 
response to events may be ordered 
according to the most critical first 
sequence: This sequence would place 
the most critical knowledge in the 
beginning of the oultline. 



Example Figure 2 provides an abbreviated e>:ample of the^ 

Generic Knowledge Outline for the systems training 
component of SONGS Operations Training. 
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Dascrlption of the Process Steps 



I. 



2, Develop the Knowledge Model Outline, continued 



General Overview of the System 

A. System purpose/ function 

B. System flow path 

C. Theory of system operation 
Design basis 



Detailed Description of the System 
A, Major system components 

1. For each system component: 

a . Name/Type 

b. Location 
c* Function 

d* Principle of operation 

e. Power supplies (normal/backup) 

3. Controls and Instrumentation 

1. For each control i 



NaTne/Type 

Loaation 

Function 



Principle of operation 
NoETnal/Abnormal modes 



For each Instrument i 




Name/Type 
Location 

Measurement location 
Function/Parameter mea 
Principle of operation 
Setpoints/Alarms 



sured 



Figure 2, Generic Knowledge Outline 
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Description of the ProcesB Steps 



Step 2^ Validate the Knowledge Model with Subject Matter 
Experts 



Introduction This step involves the first validation of the 

Knowledge Model. The purpose of this 
validation is to correct any slgiiificant 
defects in the Generic Knowledge Outline 
before further development of the model 
occurs * 



Steps To validate the Knowledge Model with subject matter 

experts (SMEs) , complete the following steps i 



Step 


Action 


1 . 


Develop the val idat ion questionnaire . 

The validation questionnaire should 
rate each knowledge listed on the 
Generic Knowledge Outline by its 
relative importance to j ob 
performance (See Figure 3) . 




Administer the questionnaire . 


3. 


Process the results . 

Determine the score for each item 
rated. Select or deselect the items 
based on a predetermined cut-off 
point . 


4. 


Finalize the Generic Knowledge Outline 

Based on the items scores, select or 
deselect the items. 

Evaluate any items collected as 
anecdote 1 data for add it ion to the 
outline . 
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Description of the Process Steps 



Step 2, validate the Knowledge Model with Subject Matter 
Experts , continued 



System: 
Names 



PLANT SYSTEMS TOIOWLEDGE ANALYSIS WORKSHEET 

Job/Position I NPEO ACQ CHS SS 

_____ Unit (a) I 1 2/3 



Datei 



Use thr information in the table below to rate the importance 
of each of the knowledge categories listed below to 1ob 
performance* ■ 



0 Not Applicable 

1 Nice to Know .Info 

2 Relevant to Job 



Knowl edge Category 



3 
4 



Important to Job Performance 
Essential to Job Performance 



Importance to 
Performing Job 



Generic 
Object. 



Generial Overview of the 
System 



A, 


System purpose/ function 


0 


1 


2 


3 


4 


B, 


System flow path 


0 


1 


2 


3 


4 


C. 


Theory of system 
operation 


0 


1 


2 


3 


4 




Design basis 


0 


1 


2 


3 


4 



II. Detailed Description of 
the System 



A. Major system components 



Figure 3, Generic Model Outline Validation Questionnaire 



ERIC 



199 



IV. A. 1 .19 

Deacrl^ptlon of the Process Steps 

Step 5, Convert Topi^^s to Knowledge and Skill Statements 

Introfl^^tlon KnowlaMdge and Skill statements describe how 

job iamcumbents use the topies aontained in the 
Qeneyj — c Knowledge Outline* 



Applic^p^tlon Knowl^— dge and Skill statements provide a frame 
refesf^^nce for the topics contained in the 

Ganetl a Knowledge Outline* In developing the 

Knowl^^dge and Skill statements^ saoh knowledge 
item from the outline is sKpanded to describe 
its ap^plication on the job. The final 
stateittments cover two levels of information i 

fe« knowl edge , 

skills* 

Note I Skills include both psychomotor and 
cognit^ive processes. 

Exams 1 - es 

I^nowledge — The formula for calculating 
Leak Rate* 

S^lcllls 1, Use of a calculator ^ 

2 . Performing the Leak Rate 
calculation. 

Figure 4 provides examples of Knowledga and 
Skill ^^tatements for the Generic Knowledge 
OutHn»« used in operations training systems 
lesson plans at SONGS. 



2 JO 



* . L 



DesGription of the Process Steps 
Step 5, Convert Topics to Knowledge and Skill Statementi 



!• General Overview of the System 

A* Systein purpose/ function 

Remember Pacti State the purpose of the 

system. 

List the system's functions. 

B. System flow path 

Remember Faot- Draw the system flo^ath. 

Theory of system operation 

Ramember Principle i Explain the theoretlaal basis 

of system operation. 

Use Principle I Explain how the theoretical 

basis of system operation 
affects system, performance, 

Use Principle: Use the theoretical basis of 

system operations to predict 
(evaluate^ diagnose^ etc. ) 
system performance, 

D* Design basis 

Remember Facti state the design basis 

criteria. 



Figure 4, Sample Knowledge and Skill Statements 
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Dascription of the Process Steps 
Step 6, Validate the Knowledge Model with Job Incun^ents 



Purpose Since the Knowledge Model serves as the foundation 
for the later steps in which lesson objeotives are 
developed, it must accurately reflect the ictual 
job. 



Method Validating the Knowledge Model involves the 
completion of the following steps; 



Step 


Action 


1 . 


Assemble experienced job incumbents . 


2. 


Review each of the Knowledge Model's 
Knowledge and Skill StatementB . 


3. 


Solicit comments and suggestions from the 
goup regarding the validity of the 
Knowledge and Skill Statements. 

Noie: A useful test of Knowledge and Skill 
Statement validity Is to collect examples 
of their application to the job. 

A Knowledge and Skill Statement that matches 
the job will ellct plausible examples. 
One requiring revision will be difficult 
to generalize through the use of examples. 


4.: 


Resolve the comments and obtain group 
consensus , 


s. 


Revise and finalize the Knowledge Model, 
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Description of the Process Steps 



Step 8, Develop Generic Objectives for Training Programs 



Definition 



Generic Objectives are statements which link 
the Knowledge Model to the training 
raquirements of the actual job. 



Derivation 



Generic Objoctivas are derived from the 
Knowledge and Skill Statements. For each 
statement, objectives are written which 
describe how the statement applies to a 
specific position. 



Generality 



Generic Objectives expand on the Knowledge and 
Skill Statements by adding content and 
behaviors which are appropriate for the 
position being trained. 

In many applications of generic modeling, one 
model is developed which covers several 
positions. For each position covered by the 
model, a set of Generic Objectives are 
developed from the Knowledge and Skill 
Statements. 



The Knowledge Model developed for SONGS 
Operations Training described the skills 
and knowledge required by these 
positions; 

• Nuclear Plant Equipment Operator, 

• Assistant Control Operator. 



Example Figure S shows examples of position specific 

Generic Objectives written from a common Knowledge 
and Skill Statement. The Generic Objectives were 
developed for the SONGS Operations Training 
Program. 



IV. A. 1 .23 

Description of the Process Steps 

Step Bf Develop Generic Objectives for Training Programs, 
continued 

Knowledge and Ski ll sta tement 

Explain the consequences of system failure. 

Nuclear Plant Equipment Operator Generic Objective 

Explain the consequences of a failure 

on the system. 

Assistant Control Operat or Gener ic Objectives 

Explain the consequences of a ^ failure on 

the systein. 

Predict the expected manipulations resulting from a _______ 

failure on the system. 



Given the following transient^ SKplaln the expected trends 
for the following System parameters i 

Figure 5^ Sample Generic Objectives 



Application Generic Objectives are used as templater from 

which lesson specific objectives are written. 
As written^ Generic Objectives include the 
content and behaviors appropriate for the 
position being trained. 

Still missing, however, are the conditions and 
standards* These are added in step 10, when 
the Generic System Objectives are modified to 
reflect the specific system being taught. 
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Description of the Process Steps 
Step 10^ Write Lesson Specific Objectives 



Introduction In step 10, the lesson plan author uses the 

Generic Objectives as a teinplate from which 
lesson specific objectives are written. 



Steps The Generic Objectives are translated into lesson 

specific objectives by completing the following 
steps . 



Step 


Action 


1. 


Review the Generic Objectives for their 
applicability to the topic being taught. 


2. 


Rewrite the selected Generic Objectives 
into lesson specific objectives by adding 
conditions and standards which are appro- 
priate for the topic being taught. 


3, 


Complete the Lesson Specification fn^^ms 
for the lesson specific objectives (See 
Figure 7) , 



EKample Figure 6 shows examples of lesson specific 

objectives developed for SONGS Operations Training 
Program from their parent Generic Objectives, 
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Description of the Process Steps 

Step 10^ Write Lesson Specific Objectives 

SYSTEM FLOWPATH 

Generic Obiectiva 

Depcribe the flowpath through the system. 

Systaig Specific Lesson Objective 

Describe the flowpath of power to the RCP buses during 
shutdown/ start-up and power operations^ including all 
transf onners used in their most preferred order. 



SYSTEM OPERATION 

Generic Objective 

Predict the Gonsequences of (component name or system 
name) failure on the system. 



System Specif ic Lesson Objective 

Explain the consequences to 6.9 KV System opt^^^tions 
resulting from a failure of the following compcn^ntsi 

m Unit or Main Transformer 

• Reserve Auxiliary Transformer (same unit) 

• Reserve Auxiliary Transformer (different unit) 



Figure 6, Sample Lesson Specific Objectives 
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Procees Control Methods 



Introduction The psroGess associated with developing and 

impleinanting generic modeling requires a 
system of controls to ensure consistent 
results* 

The controls usually involve the following 
activities: 

e collecting feedback on the accuracy and 
validity of the Generic Objectives, 

• document control, 

• user training, 

• product review and approvals* 



Collecting Feedback 



Background Although the Generic Objectives are written In 

step 8, their development is really an 
iterative process which depends on subsequent 
revision. As the experience base grows ^ areas 
missing in the original Knowledge Model will 
become evident by the lack of Generic 
Objectives addressing those topics, 

To correct this situation. Generic Objectives 
must be written which cover the missing 
topics. Revising the Generic Objectives list 
should be accomplished by following the same 
controlled process as used in the initial 
development. 
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Process Control Methods 



Collecting Feedback, continued 



Revising To revise the Generic Objectives, follow the 

the Generic steps in the table below. 
Ob j ectives 



Step 


Action 


1 . 


Collect feedback from the Generic 
Ob J ective Users , 

Feedback should be speci f y the types 
of content and behaviors missing from 
the existing Generic Objectives, 




Assemble a SME review committee to 
determine che topics to be added to 
the Generic Knov/ledge Out 1 ine . Refer 
to Step 2 of the Process Flowpath. 


3. 


Add the required topics to the Generic 
Knowledge Outline . 


4 . 


Write Knowledge and Skill Statements 
for each topic added to the Generic 
Knowledge Outline 


5. 


Validate the new Knowledge and Skill 
Statements . 


S. 


Write Generic Objectives for each of 
the new Knowledge and Skill 
Statements . 


7. 


Val idate the new Generic Ob j ectives 
against the identified def icienci^-si , 


8, 


Revise the current Generic Objectives 
list and distribute to the user. 
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Process Control Methods 

Document Control 



Background Step 8 in the process for revising the Generic 

Objectives requires the new Generic Objectives 
list to be distributed to the user, A 
critical point of process control occurs in 
this step. 

Distributing successive revisions to the 
Generic Objectives list to the users may 
result in confusion, A method should be 
developed which addresses the problems 
associated with the users retaining and using 
outdated revisions of the list. 



Example During development of systems lesson plans for 

SONGS Operations Training^ the following document 
control methods were used; 

• listing of date and revision numbers on the 
cover page^ 

• posting the date and revision nuit±>er of the 
current list in a specified location, 

• controlled distribution of the revised lists. 

Note I As each successive revision to the 
list was distributed, the previous list 
was collected and destroyed* 
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Process Control Methods 



User Training 



Background Training the users is an important part of 

ensuring consistent and accurate application 
of the Generic Objectives. For many users, 
the Generic Objectives provide their first 
eKposure to a controlled systematic 
development process. As a result, the new 
process is often received with considerable 
anxiety. 



Training new users on the proper application 
of the Generic Objectives provides the 
following benefits i 

• anxiety reduction, 

• understanding of expectations, 

m delineation of responsibilities , 
0 consistency of resulting products. 



Training may be accomplished by using several 
different methods- At SONGS, we used the 
following I 

• classroom instruction , 

• On-^The-^Job training (OJT) . 

Classroom instruction was used for situations 
involving more than four trainees, 

OJT was selected for situations involving one 
to three trainees. 



By formalizing the training^ additional 
benefits may be realized. At SONGS, 
successful completion of the training 
satisfies the requirements for the Lesson Plan 
Development portion of the Instructor Training 
Program . 



Benefits 
of Training 



Implementing 
the Training 



Additional 
Benefits 
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Process Control Methods 



Product Review and Approvals 



Background 



A final step in the procsss of ensuring 
consistent and accurate use of the Generic 
Objectives involves a two phase review. This 
review consists of an instructional check 
followed by a technical check. 



Instructional 
Review 



During this activity^ a check is made of the 
author's use of the Generic Objectives by an 
instructional technologist who evaluates the 
objectives from a process perspective based on 
the foil owi ng i 

m accuraGy of the content and behavior 

translation from the Generic Objectives 
to the lesson objectives, 

• completion of appropriate docuinentatlon, 
m construction of the lesson objectives. 



Technical 
Review 



During this activity, a check is made of the 
author's use of the Generic Objectives by a 
senior instructor/sub ject matter expert who 
evaluates the objectives based on the 
f ollowlngi 

• accuracy of the technical content, 

• completeness of technical content. 
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Process Control Methods 



Product Review and Approvals^ Gontinued 



Forms An important part of the review and approval 

process involves consistent inputs and outputs . 
The us© of standardized forms aids in collecting 
the same infomation from all authors* 

At SONGS; lesson specific objectives are written on 
Lesson Specification Forms, The Lesson 
Specification Forms pr jvide the following 
information i 



• the lesson specific objectives (developed from 
the Generic Objectives) ^ 

• the Generic Objective from which the lesson 
specific obj ective was derived, 

• test item number (s) for the lesson spacific 
objective^ 

applicable references for the lesson specific 
obj ective ; 

• suggested instructional strategy for the 
lesson specific objective. 



Benefits Lesson Specifications which include the above 

of Lesson information provide the users with three 

Specifications types of important information; 



• the material to be reviewed and approved, 

• an audit trail which documents the link 
between the Generic Objectives (task 
analysis) , lesson objectives^ test items 
and instructional strategies , 

m the micro design of the training program 
as represented by the instructional 

strategies , 



Example Figure 7 provides and example of the Lesson 
Specification Form used at SONGS, 
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Process Control Methods 

Product Review and Approvals, continued 

Lesson Specification Fomm 
Enabling Objective 

Lesson Title i 

Enabling Objective # i (Lesson Specific Objective text 

is written here) 

Classification I 

Related 

Objective 

Number: 

Test Formats Short Answer ( ) Fill-in ( ) m/C ( ) 

T/F C ) Performance ( ) 

Helated Test 
Item Number: 

Related OJT 
Qual Card Item: 

Related References* 

STRATEGY 
Main Idea: 
Keypolnts : 
Presentation : 
Examples : 
Practices I 
Feedback: 



Figure 1 , SONGS Lesson Specification Form 
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FUTURE APPLICATIONS OF GENERIC MODELING TO TRAINING 



Applications This paper has focused on the use of generic 
Within ISD models in the analysis of tasks for the 

Operations development of training materials. There are^ 

however^ other potential uses for this concept 
within the instructional systems development 
process i 

• developing test items ^ 

• identifying appropriate instructional 
strategies * 



Future Generic models have potential application to 

Applications other areas of training within the nuclear 
to Training industry^ Those situations can be 

characterized as follows s 

• tasks that can be logically grouped into 
categories that share a common attribute; 
1 . e p ^ system , component , duty area or 
work unit; 

• the task pool exhibits a set of 
informational data points or topics that 
are common to all tasks in the category; 

• task performance within a given category 
has a common set of learning or knowledge 
prerequisites * 

There are several areas of nuclear training 
where the generic model concept can be applied 
with potential success . These include : 

• Operating and Administrative Procedures 
tasks ; 

• Systems maintenance tasks ^ electrical, 
mechanical / and electronic components 
troubleshooting, repairing, diagnosing 
and replacing tasks ; 

m Quality Control inspection tasks. 
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SUMMARY 



Benefits Use of generic models at SONGS gave produced 

several important benefits to the development of 
Operations Training Programs. These benefits arei 

# increase in the level of Information quality 

and task consistency of both task analvsis and 
learning objectives; 

9 reduction in the time required to complete 

task analysis and objectives development by an 
estimated 50 percent; 

o improved consistency in the development of 
lesson plan elements and in the process for 
producing training materials. 
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Figure 1^ Generic Modeling Process Flo^^ath 
Figure 2, Generic Knowledge Outline 
Figure 3, Generic Model Outlina Validation Questionnaire 
yxgure 4, Sample Knowledge and Skill Statments 
Figure 5, Sample Generic Objectives 
Figure 6, Sample Lesson Specific Objectives 
Figure 7, SONGS Lesson Specification Form 
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USING A MICRO COMPUTER BASED TEST BANK 



Reed T, Hamel 



ABSTRACT 



Utilizing a rnicro computer based test bank 
offers a training department many advantages and 
can have a positive impact upon training proced- 
ures and examination standards* Prior to data 
entry. Training Department management must pre- 
review the examination questions and answers to 
ensure compliance with examination standards and 
to verify the validity of all questions* Manage- 
ment must adhere to the TSD format since all 
questions require an enabling objective numbering 
scheme* Each question is entered under the 
enabling objective upon which it is based. Then 
the question is selected via the enabling object' 
tive. This eliminates any instructor bias 
because a random number generator chooses the 
test question* However ^ the instructor may load 
specific questions to create an emphasis theme 
for any test* The ^^^mination/ answer and cover 
sheets are producea ^ printed within minutes. 
The test bank elimin^u i the large amount of 
time that is normally required for an instructor 
to formulate an examination* The need for 
clerical support is reduced by the elimination 
of typing examinations and also by the sof tv/are ' s 
ability to maintain and generate student/course 
lists^ attendance sheets |. and grades* Software 
security measures limit access to the test bank, 
and the impromptu method used to generate and 
print an examination enhance its security* 
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The primary purpose of test bank is to utilize a 
micro computer to generate an examination within a very 
short time period* This is accomplished by the soft- 
wa^^^'s hierarchical design of one (1) course title linked 
to many enabling objectives^ and many enabling objectives 
linked to even more test questions* (For example, the 

Authority's goal is five (5) test questions per each 
enabling objective*) After selecting an enabling objecti^ 
test bank allows the instructor to either select the test 
questions randomly or to permit him/her to pre-review the 
questions in order to make a selection. 

Before accepting test questions into any bank^ the 
questions must be reviewed to ensure compliance with 
plant Training Systems Development (TSD) policies and 
procedures* All test questions require an enabling 
objective. Licensed operator questions follow a special 
procedure noted in Table 1, (See AppendiK.) 

This procedure includes operations staff input to 
the review cycle* The loop ensures a high degree of 
validation and is a means to provide direct feedback 
to training program content* Licensed operator enabling 
objectives receive a point value. This ensures that all 
questions linked to an enabling objective have the same 
point value. Another advantage of the review cycle is 
the consensus attitude embodied in the cycle. Review 
participants are asked to read the question and complete 
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a short questionnaire^ assign a point value, and draft an 
answer* (Table 2^ See AppendiK.) The Training Department 
evaluates the responses and strives to reach a consensus , 
Once achieved^ the question can then be entered into the 
bank. 

The micro computer based test bank was developed 
in-house by New York Power Authority staff f It is operated 
by the instructor initiating the examination* It is menu 
driven with each menu level password protected. 
(See Table 3 . ) 

Table 3. Test Bank Master Menu 

TRAINING DEPARTMENT 
MASTER MENU 

1. Test Bank Data Entry Menu 

2. Test Bank Report Menu 

3 . Student Information 

4. System utilities Menu 

5. Supervisor Menu 

6. Exit to Operating System 

Please Enter Your Selection \ i 

Please Enter Valid Password. 

Test Bank has three major functions. The first is data 
entry and editing of test questions. Second is the 
examination output, and third is student information 
and grades data. 

* Test bank utilizes Dataflex by Data Access Corporation*, 
Miami, Florida 
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For data entry ^ the course description and 
enabling objective are requested, (See Table 4,) 

Table 4, Test Bark Question and Answer Data Entry Screen 

TRAINING DEPARTMENT 
QUESTION AND ANSWER DATA ENTRY SCREENS 

1, Please enter the Course Title i 

Depress "F2"'Flnd Course 

2. Please enter the Course Number & E.O^i 

(No Spaces) ^ 
Total number of questions presently for this E.O, : : 

DEPRESS "F8" FOR HELP 

Once entered the bank is ready to accept either a multiple 
choice or essay question* This fixed record length pro-- 
gram will hold up to 1050 characters for the question and 
over 1700 characters for answers. Essay is used for 
typical essay questions, true^false, f ill^in-the-blanks 
and matciiiiig columns, (See Table 5,) 

Table 5, Test Bank Question and Answer 
Data Entry Screen II 

Total number of questions presently for this E,0, i 
Question Type i (M)ultiple Choice or (E) ssay * ' ]~ — ^ 



DEPRESS ENTER TO ADVANCE TO THE ANSWER SCREENS 
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Test bank has no graphics capabilities. Therefore, line 
diagram or drawing answers must be noted by reference. 
All test questions are saved in ASCII Code, This allows 
most Greek letters and mathematical symbols to be used. 

Of course^ the real value of this micro computer 
based test bank is the ease and efficiency of initiating 
an examination. A specific examination can contain up to 
75 test questions and is used for daily or weekly tests. 
The licensed oparation version also permits the instructor 
to prepare mock "NRC License Operator*- examinations via 
the eight (8) NCR categories (Table 6, See Appendix,) 

For either type of examination^ the instructor must 
answer the prompts * These include today's date^ test 
date^ course title, test number and number of test 
questions. (See Table 7f) 

Table 7 . Test Bank Initiate Examination Screen I 

TRAINING DEPARTMENT 
INITIATION OF AN EXAMINATION 

Today's Date: 1/29/87 

Please enter the following information i 

Course Titlei Atomic Absorption i 

(Depress 

Date the test is to be administered! 1/29/87 s 

Section Number (if more than one section being taughtTi^^ j 

Test Number i 2 i 

Number of required questions for this test i 3 i 

Remarks I ' i 



DEPRESS RETURN TO ENTER DATE-TAB TO RETURN TO PREVIOUS 
WINDOWS DEPRESS "ESQ" TO EXIT 

DEPRESS "FB" FOR HELP 
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Then the course enabling objectives are displayed, 
(See Table 8,) The instructor chooses from this list 
until the niunber of questions selected equals the number 
requested. 

Table 8. Test Bank Selection of Enabling Objectives Screen 2 

TRAINING DEPARTMENT 
SELECTION OF ENABLING OBJECTIVES 

Today's Datex 1/29/87 

Course Titles Atomic Absorption Numbers CSPT02 



EO s for this course I Questions Available 

1 2.2,1 3 

2 2*2,2 5 

3 2,2,3 5 

4 2,2,4 5 
NO MORE EO's 

To select an EO enter a nmnber (1-10) ( ) 

DEPRESS "FIO" FOR MORE, DEPRESS TO START OVER 

DEPRESS "FS" FOR HELP 

After selecting the enabling objective, the instructor 
can choose to select a question randomly or preview the 
first 70 characters of the question* (See Table 9,) 
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Table 9. Test Bank Select Question Screen 3 

TRAINING DEPARTMENT 
SELECT TEST QUESTIONS RANDOMLY 

Course Titles ATOMIC ABSORPTION Numben CSPT02 

Choice Selected E,0. Questions Available 

4 2,2.4 5 

You have requested 5 questions for this test. 0 have been 
selected so far. 

How many questiot^s do you want selected from this enabling 
objective ? 

Do you wish to pre^review Test Questions for the selected 
E.O,? (Y/N) 

DEPRESS FOR HELP 

When the number of questions selected equals the 
number requested^ the test automatically prints one copy 
of a pre-format examination, a separate answer sheet and 
a cover sheet* All printing includes headers, spacing, 
question numbers and page numbers, this whole process, 
depending upon the number of questions chosen and how 
much pre-^review takes place ^ can literally take only 
minutes. The number of test questions requested does not 
inhibit the programs speed • The test is archived to a 
file and the requisite examination copies are photocopied. 
(Tables 10, 11 and 12, See Appendix.) 



223 



IV. A. 2. 8 

The third major area is student information, it 
compiles student data such as name, course, social security 
number, title and test grade. Various reports permit 
tracking student course enrollments^ grades and final 
average grade . 

In summary, the positive impacts of the micro 
computer based test bank are as follows: 

a. Management review cycle and consensus, 

b. Significant time reductions in initiating 
examinations. Typically a reduction from 
two (2) hours to fifteen (15) minutes. 

c. Ease of operation. 

d* Elimination of clerical typing. 

e. Security considerations through computer 
logins, password protection, and impromtu 
test generation. 

f. Training aaministration support through 
course reports, student tracking and 
grade matrix capabilities. 

g. Database compatability for both weekly 
examinations and mock "NRC Licensed Operator" 
examinations . 

h. Program's flexibility to operate in almost 
any test environment utilising an TSD format. 



EKLC 
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FLOW CHART OF TEST QUESTION REVieW CYCLE 



SELECT 
QUESTION 
BY TRAINING 




Y 

E 
S 



REVjEW & 

VERIFY 
QUESTION 




Y 

E 
S 



FIN^ 

a FO 

QUE^ 


lLIZE 

RMAT 

3TION 


'I 


f 


ENTER 
INTO 
BANK 







STOP 
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Table 2. Test Bank Questionnaire Used At 
James A. FitzPatrick Nuclear Power Plant 

The following guidance is offered in your evaluation of 
the questions and answer(s)^ 

1. Review the question BEFORE looking 
at the answer provided. 

(Y) (N) (N/A) 2, Does the question adequately tost the 

knowledge requirements of its corres- 
ponding Enabling Objective? 

3* Does the question solicit a specific 
response which is a **required" 
knowledge or skill for your job 
position? 

4, Answer the question in your own way, 

(Y) (N) (N/A) 5, Compare YOUR answer to the answer 

key; are the same key points 
identified? 

(Y) (N) (N/A) 6p Does the point value (s) assigned to 

the answer match your eKpectations 
from the question? (An aid to 
answering this question is the 
qualification standard for your 
job position*) 

7. Make your comments in the space 
provided below, (Please comment 
to any "NO" response) , 
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Table 6, Initiate *'NRC" Examination 



Training Department 
Initiation of Operator Examination 



NRC CATEGORY 

1, Principles of Nuclear Power Plant Operation^ 
Thermodynamics^ Heat Transfer and Fluid Flow 

2, Plant Design Including Safety and Emergency 
Systems 

3, Instrument and Control 

4* Procedures = Normal^ Abnormal^ Emergency 
and Radiological Control 

5* Theory of Nuclear Power Plant Operation^ 
Fluids and Thermodynamics 

6* Plant Systems I Design ^ Control and 
Instrumentation 

7, Procedures 4 Normal ^ Abnormal^ Emergency 
and Radiological Control 

8- Administrative Procedures^ 
Conditions and Limitations 



Please Enter Your Selection : 
DEPRESS "F8" FOR HELP 
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Table 10. Sample Test Bank Examination 

James A. FitzPatrick Training Department - Examination 

Course #: CSPT02 Test Date i 01/2 9/8 7 

Section #• 1 Test #-2 

1. Plot the following data in the best way possible and 
use your plot to determine the concentration of an 
unknown with a transmittance of 231. 
Standard Cone, (ppm) % Transmittance 

0>0 100% 



2 . 



3. 



0.5 



40.2% 



11.3% 

Contrast high and low dispersion gratings and explain 
wny one IS preferred over the other. 

When creating a dispersion grating, blazing is the 
teciinique used. What is it and why is it used?" 
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Table 11. Sarnpla- Test Bank Examination 
Ml^ swer Sheet 



James A, PitzPatr^^ick Training Department 
Exarninati on Answer Sheet 

Course #i CSPT02 Test Date i 01/29/87 

Section #t 1 Test 2 

E.O. #: 2*2,1 

1* Percentage transmittal ^ce « 23, so the cone. « 
0.80 ppm , 



A ^ 


log 


(1/T) 




A ^ 


log 


(1/1) ^0.0 




A - 


log 


(1/.402) * 0. 


396 


A - 


log 


(i/ai3) ^ 0. 


947 


A - 


log 


(1/.23) ^ 0.6 


38 



Refer to Reference 1 * Supplement for plot drawing, 
E,0. #^ 2-2,4 

2* High dispersion grati^xigs have a much greater line 
density ^hich providers much greater interference 
in some arsas and muc^i greater additivity in others. 
The result is a bette^c dispersal of wavelengths with 
the high dispersion g^^ating which allows us to obtain 
narrower peaks and Im^ss interference, use larger slit 
widths which in turn ^^esults in less amplifier gain 
and better sensitivity^ through the reduction of noise 

E.O. #• 2.2.4 

3, Blazing Is the proces^s of making gratings by carving 
V-shaped grooves into a reflective surface. It 
allows the creation a dif faction grating which 

gives good accuracy a^nd dispersion, yet is relatively 
inexpensive* 
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Table 12. Sample Test Bank Cover Sheet 

New York Power Authority Examination/Quiz 
James A, FitzPatrick Nuclear Power Plant Cover Sheet 

Examination Title: Atomic Aij^orption Test Dater 01/2 9/8 7 
Examination Approval : 

(Program Administrator) 
Open Book ( ) Close Book ( ) Time Limit i 



Authorized Reference 
Material • 



Minimiim Acceptable Grade i Grade: Graded By: 



Student Data 

Name i S . S * # - - 

(Last) (First) (M,I , ) ^ ^ ~~ "~~ 

Employers Date: 

Department : 

Guidelines 

1. Remain seated and quiet during the examination* 

2. If you have any questions during the examination^ 
raise your hand* Your instructor will provide 
clarification wherever possible. 

3* You are expected to do your own work and not help 
anyone else, 

4. Use only the authorized reference material* 

5. At the completion of this examination^ you are to 
sign the following certification. 

I certify all answers contained in this examination 
are my own. In addition, I have not received nor 
given any unauthorized assistance, nor have used any 
unauthorized references* 

Student Signature : Date : 
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THE ROLE OF SIMULATOR T RAXWHg, jffjEi^t^Qflj 
TEAMWORK AND DIAGNOST tC SKlLLl 

W. E. Grlmnii 



ABSTRACT 

A review of the evolution of the contrQl fo^sEaani 
is necessary to understand team tralnitig ii^%Bih As 
control room responsibilities have itiGt^%#^d ind 
members have been added to the operatitif c^^ysj 
teamwork and strong leadership lias beCo0% ^jycial to 
the efficiency of these operating creWs* Ifi order 
to conduct effective team tralnixig iri a %j^^lited 
control room, it is essential that the f^ji^anental 
principles of role definition and cOmflioti t^#values 
are fully developed. 

The diagnostics model used to develop p^Q^leffl^ 
solving skills must be adaptable to th^ 4yA^i!ilc 
environment of the control room* OnQe 
fundamental principles of team buildiiig %tid a good 
diagnostics model are mastered , many tt#i,tii^| 
techniques using a simulator are avall^^l^g zq 
perfect the development of team buildiiig #ftd 
diagnostic skills. 



Any study of the current training techniqaes being tjiyto develop teai 
work and diagnostic skills in control room c^atgg ±B g^gsEly enhanced 
a review of the evolution of today's control toom t^^t^^ Today 's team 
training programs are largely products of trairiera ^^ti^j liiva had the op^ 
portunity to observe and work with hundreds of opere^c^ts over thousandi 
of hours during this evolutionary phase. tb%i^ trfti^%tshave been sup- 
ported by specialists in the fields of hutnan behavio^ an^ managemeiit 
techniques s and their combined efforts hmvm tesylte^ observable tm^ 
provements in the teamwork and diagnostic capabilitg^^^ of the operators 
and supervisors who have participated In the trainihg pMgrams developc 
by them. 
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Probably the earliest Idea in the movement toward team building in con- 
trol room crews was the concept of the operator and the senior operator, 
or the RO and the SRO. This early distinction seemed to provide some of 
the elements such as leadership and coordination of efforts thought to 
be needed in a successful team. In actual practice, however, the dis- 
tinction really only indicated the depth of study required for each posi- 
tion. IJhile the RO studied how to operate Bystems and monitor parameters, 
the SRO studied these RO skills and, in addition, the theory and the bases 
for actions taken and limits observed. The end result of this effort 
was essentially a control room with two operators, one of whom could ex- 
plain why things were being done. Very little actual teamwork was seen 
with this arrangement. Even though the RO and SRO may have worked well 
together, there was really no guidance, leadership, or coordination of 
effort to achieve efficiency. 

The actual difference between the RO and the SRO became even less notice- 
able after 1979 when increased emphasis was placed on training in thermo- 
dynamics, degraded core concepts, and other areas of plant physics. 
Suddenly, while the SRO was required to study more of the background 
topic, the RO found himself being trained in areas previously thought to 
be "SRO material." 

In 1981, two new concepts came into play to further complicate the struc- 
ture of the control room team. One of these concepts was the addition 
to the team of a degreed engineer to supplement the SRO in the engineering 
aspects of plant operation. This engineer, the Shift Technical Advisor, 
or STA, was potentially a very valuable resource, but to be used by whom? 
and how? The answers to these questions wsre not supplied with the STA. 

The other concept affecting the control room team was the new requirement 
that SROs receive training in the areas of problem-solving, decision- 
making, and prioritizing. This training seemed to indicate that the SRO 
was the man to whom the team could look for leadership. He would utilize 
the resources of the ROs and the STA to solve problems, make decisions. 
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and set priorities. One big obstacle to this goal was the fact that, 
typically, the SROs were chosen based on their abilities as ROs: usually, 
a good RO became an SRO, Often this new and poorly defined role was not 
a comfortable move for the good operator and the new SROs were sometimes 
slow to become leaders, problem-solvers j declsion^makers , and priority- 
setters . 

Another problem started showing up in the 1980 's. The established SROSj 
some with many years of experience , began to be left behind as the new 
SROs continued to study the physical sciences in their training programs- 
AlsOi the older SROs were faced with learning new ways to solve problems , 
make decisions , and set priorities and sometimes couldn't understand why 
their experience in these areas didn't seem to count. Why weren't the 
techniques that they had developed over years of experience acceptable? 

The general training situation that existed through 1985 was as follows^ 

1* ROs and SROs generally trained together s although not necessar- 
ily with their own teams* 

2* STAs typically did not train with the team that they supported, 

3, SROs continued to be trained in the team-leader skills of 

problem-solving p decision-makings and setting priorities, but 
these skills were not really catching on; most of the team 
moimbers elements required for successful implementation of 
these skills were missing* 

4* All the elements for a successful team consisting of supervisors, 
operators p and technical resource people were in place but had 
not really been brought together. 
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In 1985, the team building and diagnoaticB movement began in earnest and 
genuine commitments were made. INPO provided some good fundamental 
guidance in TQ503 — a team building plan using a four-module format. 
The four modules were basically communications, dlagnoatica, case study, 
and application; this last module was to be set in a simulated control 



room. 



The remainder of this paper describes General Electric 's Nuclear Training 
Services experience in assisting several utilities in their team training 
efforts . 



The first teams to experience team training based on the first three 
modules of INPO 'a TQ503 consisted of operators, aupervlsors, and sometimes 
STAs. Bringing these team elements together for the purpose of exploring 
team building and diagnostics was very enlightening with regard to some 
of the problems that needed to be overcome. One of the very first obser- 
vations was that the individual team members did not really know what was 
expected of them, nor did they know what they could reasonably expect 
from the other team members. Another observation was that, in some cases, 
the stresses resulting from undefined team responsibilities were causing 
areas of personal conflict. These personal conflicts could not be re- 
solved satisfactorily within the present team structure. 

These two observations prompted the first modifications to the team 
building and diagnostics training. The original training program seemed 
to have been developed with too many assumptions; the program needed to 
start with some even more fundamental concepts and exercises than hitherto. 

The first Improvement consisted of exercises and seminars aimed at de- 
fining roles. The lecture format was deliberately avoided and seminar- 
exercise techniques were used to turn role definition into a set of 
solid expectations that the operators, supervisors, and technical advisors 
could depend on. Each individual team member was asked to describe hia 
own perception of his role and his perception of other member's roles. 
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Feedback from the other members was then solicited^ Eventually, the 
perceived and real expectations inatched , Thus, these seminar-exercises 
relieved the stress of unknown responsibilities and tore down the initial 
obstacles to effective team building; they also formed the basis for in- 
dividual team member accountability* 

The next step was to work on establishing team values* Without cominon 
values, the energy of a team Is diminished through dispersion. It was 
observed in initial groups that, even though all the individual values had 
the same name of "operate the plant safely," each team member typically 
had a different idea with regard to achieving this goal* By establishing 
conunon values ^ a team culture was formed whereby the common goal overrode 
the personal preferences of the individual team members. The crew members 
were actually beginning to form a team. 

The title of this paper is "The Role of Simulator Training in Developing 
Teamwork and Diagnostic Skills," By now, you are probably wondering if 
there is a role for the simulator in developing ' team building and diag-^ 
nostic skills. The truth is that until a disciplined team, with cominon 
values, with leaders who hold the team members accountable for their 
actions and responsibilities, and with members who know what they can 
expect from others and what is expected of them, enters the simulator, 
team training may take place but team building cannot. The stresses 
resulting from unknown responsibilities and conflicting goals, combined 
with members holding back their performance because there is no account— 
ability, can actually prevent a team from forming. But when the funda- 
mentals are understood and team agreement is reached ^ the simulated 
control room training becomes the team training. 

With these fundamentals established, then, team building and diagnostics 
moved into the simulated control room. The Initial experience was very 
good. Team members who had sirred the classroom experience of team 
building and diagnostics were visibly more comfortable in their roles and 
exhibited more focussed energy in their approach in controlling the plant* 
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Along with this new phase of training, however, came another observation 
that called for a change in course strategy. 

The second module in the team building and diagnostics training consists 
of diagnostics (the problem-solving), while the third module consists of 
tabletop case studies to practice diagnostic skills. The Initial approach 
used in diagnostics training was a management style problem-solving and 
decision-making technique. The technique worked very well on "paper plant- 
problems; however, the team participating in the training had a difficult 
time making the transfer from paper to the simulated control room. The 
diagnostics approach was not natural to a dynamic, operating environment 
such as a simulator. A more natural problem-solving process was needed 
in order to produce a functional problem-solving team. The source for 
the new diagnostlos system came from observation of the operating crews. 
It was found that the strongest problem-solving crews used a process of 
Iterations on discrete pieces of information made in an orderly fashion. 
Based on this observation, an algorithmic approach to problem-solving was 
Implemented in the team building and diagnostics training. This approach 
proved to transfer more naturally into the dynamics of the control room. 

With the team culture developed and established and the algorithmic 
diagnostic model introduced, polished simulator team building and diag- 
nostics training could finally be conducted. Prior to each simulator 
session, a review of the classroom concepts of team values, role defini- 
tions, diagnostics, and communications is held. These reviews serve to 
reinforce concepts and focus each team member's attention toward the 
team building and diagnostics portion of the simulator training. The 
review sessions include additional tabletop practice and experience report 
case study. The topics are very well received by the members of the 
operating teams. In fact, it seems that they welcome the opportunity to 
begin their simulator training sessions with a "team warmup" in which 
they can reestablish and reinforce the team culture development and review 
the common team values. 
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Following the classroom "warmupi 
of the simulated control room. 



" the training is moved to the environment 
Several approaches are utilized depending 



upon the particular needs of the planers Operations Department. Once the 
training techniques are defined ^ they remain constant for all crews of an 
Operations Departme 

One training techn.,,^..u uhat has become fairly common is the directing of 
all trainer-to-cre^^ questioning through the shift supervisor* The goals 
of this technique are to strengthen the supervisor in the eyes of the 
team members and to provide the supervisor with extensive practice in the 
use of his team resources* The rules of this approach state that the 
trainer, in his questioning of the teatnj will direct all questions to 
the team supervisor* The team supervisor is^ in turn 5 free either to 
"take the shot hJjnself" or to seek assistance in answering the question 
from any chosen member or members of the crew* 

Besides the stated goals , this technique has other advantages. The 
trainer can detect under -utilization of valuable team resources ^ such as 
the STAp or over-utilization of the strong team members* The trainer can 
then provide feedback to the supervisor to help djiiprove his resource 
utilization. It should come as no surprise that the more experienced 
supervisors take to this technique better than the newer supervisors* In 
fact, some of the more experienced supervisors will pose additional ^ 
followup questions to their team to ensure that an area is solidly covered. 

Another technique that has routinely been used is the technique of the 
team self --critique , In this strategy ^ the trainer's critique notes are 
supplied to the supervisor , who reviews and discusses them with the team. 
This technique stresses accountability for individual actions or mistakes. 
It is easy for the team to see that individual actions can greatly affect 
the overall team performance. The self-crit Ique concept can be likened 
to the idea of reviewing the game film and it has the potential to be an 
even more powerful tool if a video recording of the performance is 
available. 
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An overall evaluation of the team training experience by both trainers 
and crew members indicates that the initial phases of team training have 

been successful and that this training has produc-d observable improvements 
in operating crew perforinance as a team. A review o£ team critiques 
indicates a definite Improvement in the teamwork and diaanostlc abilities 
of the crews during exercises stressing normal and minor off -normal 
activities, but a possible decline In these skills during uicreasingly 
complex abnormal situations. This last observation provides trainers with 
the continuing challenge of further improving the development of team 
building and diagnostic skills. 
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□SB OF VIDSQ TAPING DURING SIMULATOR TRAINING 



Michael Helton 
Phillip Young 



ABSTRACT 



The use of a video camera for training is not a new idea and is 
used throughout the country for training in such areas as computers/ 
car repair/ music and even in such non— technical areas as fishing* 
Reviewing a taped simulator training session will aid the student in 
his job performance regardless of the position he holds in his 
OfTganization. If the student is to be examine on simulator perfor-- 
mance/ video will aid in this training in many different ways. 



INTRODUCTION 

Until recently/ simulator training was done primarily by instructor to 
student interface only* The student would perfomi and/or respond to a 
casualty and the simulator instructor would critique the evolution- 
Now the use of a small/ self-contained video camera for simulator 
training has been successfully used at various sites for improving tne 
method of training personnel. The camera critique is effective in 
improving instructor/student feedback/ resulting in more highly skilled/ 
safer operators which are essential to the nuclear ^wer generation 
industry, 

HISTORY 

Simulator training for operators begins by performing a plant startup 
with ascension to 100% power/ and then a complete shutdown* This 
intense training program cavers a period of several weeks. During the 
training sessions/ inalf unctions (casualties) are introduced beginning 
with the least severa (l*e. pump trips/ failed valves/ etc) and 
progresaing to the most BmmrB (i.e. steamline brsak/ loss of feed- 
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water/ etc.). The worst case malfunctions result in implementation 
of the site Emergency Plan and are normally taught in sessions of 
approximately 45 minutes to 1.5 hours in duration. 

Following each training session/ the simulator instructors critique 
crew and Individual student performance. The critique includes/ but is 
not limited to, student verbal response/ physical actions taken/ 
sequence of events, what was observed versus what happened/ and overall 
performance. 

It is not unusual for the critique to lead to a discussion of whether 
or not a student actually did as the instructcrs said or if a problem 
was overlooked or the session lasts as long as it seemed to the students 
Replay or backtrack features of a simulator are sufficient for some 
types of critiques but they do not show ntudent response or the actual 
elapsed time of a training session. During final training sessions, a 
video recording can be used as an invaluable training aid. 

USES 

Replaying a recorded simulator training session as soon as possible 
after the session has ended not only reinforces the instructor critique 
but also allows the student to learn from his errors almost immediately. 
He can see for himself any needs for improving teamwork, comnunications/ 
control room inanipulations, or any other areas discussed with the 
instructors. A critique of a training session by viewing a video 
recording benefiits the instructor as well as the student by improving 
his simulator instruction technique, the quality of student feedback, 
and instructor to student relations. 

Recording mid-course and/or end of course audit exams is another 
valuable use of the video camera as a training tool, in that it aids 
the student in improvinq his performance. The recording indicates his 
weaknesses in exam- taking technique under stress/ points out errors 
in exam protocol/ and enables him to take future exams with greater 
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confidence by helping him overcome these piroblem areas. The video 
replay shows weaknesses in performance/ poor habits / and mistakes thus 
allov?ing both the student and instructor the opportunity to see what 
really occurred. Usually/ the replay emphasises to the student that 
the actual elapsed time of the session was much longer than it seemed/ 
and allows the response to certain problems to be thought out* 
Realization of the "real time" could r^uce the chance of making 
serious mistakes not only in the simulator but/ more importantly/ in 
the real control room by showing the operator that sometimes an 
immediate response io not always the desired response. 

A replay can also reinforce an instructor's comments on previous 
critiques about a student or crew problem/ as shown by the following 
examples s 

1. During a taped audit exam/ a student is irritated by questioning 
and constantly evades the examiner* During the critique following 
the exam/ these observations are pointed out to him, and are 
strongly denied. After reviewing the recording/ the student and 
his crew agree that the examiner's observations were correct. The 
crew also notes areas of weakness in teth individual and team 
perforincince* It should noted that the critique of the video 
recording may also prove the examiner/Instructor to be incorrect 
in his evaluation. 

2, One crew has difficulty developing as a team/ i.e. poor corranuni-- 
cations/ poor use of procedures^ weak team ability/ etc. 
Following several v/eelcs of training with the instructors attempting 
to correct the problems/ it is decided to use video replay* The 
video replay fails to help/ so another crew/ which has developed 
these skills/ is tap^* This second crev/'s tape is shovm to the 
first creW/ typically resulting in Improving the skills v/hich were 
lacking before* 
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EQUIPMENT 



Any type of self-contained rooviG camQra with a telaphoto function can 
be used. If a self'-contain^ VHS or Beta irovie camara is used/ it 
would allow viewing to be done at home, and avoid tying up valuable 
training time following the initial viewing. 

The new models are relatively inexpensive and usually don't require any 
special lighting. One problem/ however/ can be sound quality. Most of 
the self-contained cameras have an externally mounted microphone. This 
microphone receives the usual 60=cycle 'hum' from all the electronic 
equipment in the simulator. Wireless or strategically placed micron- 
phones can virtually solve this problem but adds to the expense. 

A tripod is not essential but/ if used/ should be of sturdy construction 
simple to us©/ and compact if availability of storage space is a concern 

Recently/ the use of multiple cameras is being tested at some facilities 
These cameras are program connected to the simulator computer and 
respond to malfunctions just as they occur. Again/ this is being tested 
and cost could be a deciding factor. 

If multiple/ computer driven cameras are not possible/ then the use of 
another instructor/ who knows the evoluton/ could be used as the camera 
operator. Knowing the evolution/ he Is able to direct the camera to the 
area affected by the malfunction/ and eliminate the need to aim the 
camera before the training session begins. The student Is then unable 
to anticipate the casualty from the direction the cainera is pointing. 
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ADVANTAGES AND DISADVANTAGES 
Advantages 

1, Shows strengths and we^icwses of students during svolutions. 

2, Shows the good as well as the bad habits of the students, 
3- Shows student mistakes made during the evolution. 

4, Shows any simulator problen^ affecting the evolution, 

5. Reinforces instructor coiranents made during the training and 
critique . 

5. Allows the student to observe any prablems with coriununications. 

7. Gives the student a feel for real time during the evolution* 

8, Prevents instructor/student conflicts 

Disadvantages 

1. Sound quality' is somewhat poor due to typical 60^cycle hum* 
2* Cost of setup time since the camera cannot be left out at all times* 
3* Equipment cost must be considered since budgets are of concern* 
4* Caniera fright of student while trying to perform during an eKain* 
5* Availability of non-^interfering but effective camera location* 

"^This could be an advantage by helping the student to overcome the 
uneasy feeling of being observed* 

SUra-OMY 

Just a few years ago/ reactor scrams were coimiionplace in nuclear plants 
around the country j but today/ as a result of intensive training # it 
c€in be seen that the best method for operating a plant safely and 
reliably is to operate at full load/ consistently* ^^e train the 
operators to operate in thia inanner/ ffjonitor control room parameters/ 
and respond to abnorrrial aituations indlcati?d by thi^^a paranietfsro* 
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Today, wa are moee aware of how safe, reliable and consistent plant 
operation is dependent on quality training. Public safety and the 
financial haalth of the nuclear industry depends on the training 
provided . 

It is obvious that the advantages by far axdeod the disadvantages of 
improving training techniques to produce a more highly-skilled, 
competent, and most importantly, a safe plant operator. 

The use of video recorded training sessions is and will continue to be 
a valuable training aid as long as there is a need for well trained 
operators , 
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I&C MAINTENANCE TRAINING SIMULATORS 
MEETING THE GOALS OF NUCLEAR PLANT TRAINING 



Samson Shilmover, Jr* 



ABSTRACT 



A^ we eKamine the goals of traJnlng from the point of view of the 
instructor, student, and employer, we will see that I&C maintenance 
training simulators meet or exceed those goals* 

An instructor -s overall goal in I&C training is to Impart os much skill 
and knowledge as possihlfc about the operation ^ tune up and 
troubleshooting of a piece of equipment. Through the use of a training 
simulator, the instructor has the "actual" equipTiient to demonstrate the 
principles he is trying to make, 

A student -s goal in training is to Improve his skills by learning as 
much as possible about the equipment he will be responsible for. When a 
simulator is used, a student gets hands-on time with the equipment where 
normal and abnormal conditions can be demonstrated. The students may be 
asked to perform Job— related tasks and be evaj.uated on their 
performance * 

An employer's goal is to get personnel who can quickly fit into their 
work force. When training is performed using simulators i the employee 
receives the knowledge and hands-on skills necessary to quickly fit into 
the work force. The employee also gains confidence in working with the 
equipment. Another advantage for the employer is that training can be 
perfnrmed without shutting down the plant equipment. The employer thus 
can use the trained employees sooner and mainstream them into the work 
force. 

In summary, the use of I&C maintenance simulators can greatly enhance 
the quality of training and will prove to be beneficial to the 
instructor, employee, and employer. Although other training m.ethods can 
meet some of the goals of trainings the use of simulators will decrease 
training times increase the operational skills of employees, decrease 
probability of equipment damage due to employee error* and increase 
employee productivity and safety. 
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SIMULATOR TRAINING AND SAFETY ISSUES 

"Training of personnel who operate and maintain nuclear power facilities 
is a vital part of the safe operation of these facilities."^ Assurance 
that personnel have received the highest possible level of training is 
one of the highest priorities of any Industry where personal or public 
safety is involved. In those industries where personal and public 
safety is involved, training simulators are used to assure that 
personnel rocelve the quality of training required to guaranty that 
safety standards are maintained. 

The utilization of simulators to meet learning objectives can be one of 
the beat means of assuring that critical skills can be performed safely 
and In a timely manner. When a training simulator Is tied into a course 
of learning the benefits accrue to all Involved in the training process. 
By examination of the goals of the training process, and what is 
required to effectively meet these goals, it can be clearly seen that 
the use of ti-ainlng simulators greatly enhances the training process. 

Simulation, while expensive, is regarded as the best - often the only - 
way to approach some performance training needs. In most cases, the 
expenses of simulation pay for themselves as soon as a condition arises 
where it is obvious that training alone was responsible for averting 
some serious problem. The more closely a simulator resembles the actual 
equipment, and the responses of that equipment, the greater the benefits 
will be to the training received. Pilots, at different stages in their 
training, receive Instruction in a simulated aircraft cockpit that 
looks, feels and reacts like an actual airplane cockpit. The training 
simulators utilized by the airline industry are normally exact 
duplicates of the actual equipment connected to a computer for simulated 
response of the aircraft. This allows abnormal and normal conditions to 
be evaluated for correct response without the risks of endangering 
public safety. 
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In the nuclear industryi control room operators are required to be 
certified on a simulator prior to taking over control room watches. 
Many utilities have gone to the expense of duplicating control rooms and 
require their operators to spend off-shift time performing on the 
simulator* In this way the utilities are assuring that operators can 
handle normal and abnormal evaluations without the expense of lost 
revenues of performing training evaluations on the plant* 



The advantages of using training simulators in a ISC maintenance 
training course must be eKamlned from the point of view of the 
instructors trainee * and employer. To effectively utilize a simulator 
in a course, the course must be structured around the use of the 
simulator* In the ISC training environment this dictates that the 
maximum time be spent either demonstrating or having the trainee perform 
on the simulator* Experienced ISeC and navy nuclear Instructors claim 
that early career training on a simulator has the following advantages: 



1* The trainee is quickly put at ease about 

working on the equipment ; 
2. The instructor can get a feel for the pace at 

which the course of instruction should ' ^* 

geared; and, 
3* The enthusiasm of both the trainee and the 

instructor becomes heightened* 



To an instructor, the use of a simulator in a course can be invaluable* 
Key concepts can be demonstrated or pointed out directly on the 
simulator thus reducing the amount of time required to "get the point 
across*" Demonstration and perforfflance capability is the major 
advantage to one^on'-one instruction in a course where a simulator is 
utilised. In a tutorial or small group setting, the simulator 
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instructor can provide practice, can perform formative testing, and can 
observe tool using skills to decide whether the trainees are ready to 
move to the next step in their training. The feedback responae of 
observing the trainee, while performing on a simulator . la an invaluable 
tool. It becomea readily apparent whether the key points where brought 
acroas to the traineea by observing their performance while operating or 
working on the simulator. Course pacing ia another advantage of the uae 
of simulators for an Instructor. The Instructor can develop a feel for 
"pacing" on a simulator, and can correct performance problems by further 
review and trainee practice. When trainees satisfactorily demonatrate 
their abilities to perform specific tasks the training cycle may move on 
to on-the-job training, classroom, lab. or another simulator setting. 

The use of a simulator has numerous advantages to the trainee. First, 
there Is a higher level of enthusiasm for the material to be covered. 
When a training simulator la tied to a courae of learning the trainees 
become enthusiastic about not only the performance objectives but also 
the theory of operation of the equipment. This is because adult 
learners are generally more satisfied when motor skills training and 
coanitlve training are both present In a course. Second, trainees can 
quickly build confidence in their abilities by the immediate feedback of 
performing tasks correctly on the simulator. Trainees are required to 
commit to a course of action and are able to see the responses to their 
actions. When the trainees see that they can perform successfully on 
the simulator their confidence that the knowledge and skills obtained on 
the simulator can be transferred to job related tasks is assured. 

In the risk-free environment provided by the training simulator, 
experiments can be performed and trainees will be able to measure the 
effects of their actions. This can be a major advantage especially 
where abnormal evaluations are performed (i.e. troubleshooting and 
equipment modification). Simulators can help trainees to develop a 
systematic approach to exploring the cause and effects of abnormal 
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operations of the equlpTnent, and help them establish a basis for 
evaluation of proper operation. 

On^the--job training and classroom training often can take a longer 
period of time to establish needed skills than does simulator training. 
Employees i then, are often less productive for a longer period of time. 

Dpwn=time of essential equipment can be reduced with simulator training 
since the employees will be more familiar with the proper operation of 
the equipment, and be able to isolate faults at a faster rate. Normal 
evaluation such as equipment alignment and general preventive 
maintenance will also be performed with greater accuracy and with less 
down^time of the equipment as a result of simulator training. 



CONCLUSION 

Simulators are a effective training tools, and their cost should be 
measured In terms of the consequences of inadequate employee 
performance * 

Some sources estimate that an unplanned outage costs a utility 
approximately one million dollars a day In lost revenues* The reasons 
for an unplanned outage are many, and the duration of outages varies 
widely^ To the extent unplanned outages are a result of inadequate 
training, the decision to use simulation as a method of reducing the 
number and duration of unplanned outages should be evaluated carefully* 

Employees generally respond well to simulationj and expert evaluators 
can observe not only the level of trainee performance, but whether a 
trainee exhibits any overt nervousness which may need to be addressed 
through increased practice or other means. 
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"No matter how carefully designed or how well the equipment functions, 
the engineering syatema cannot work without people... People are 
necessary to monitor, correct, adjust and take other actions to maintain 
the systams." Simulators can be a cost affective tool when used to 
train employees who perform tasks which are relied upon to keep the 
plant up and running In a safe and efficient manner. 
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INCIDENT AL INSTR UCTOR TR A iNING- 
DUKE POWER^S ANSWER TO 
PREPARE THE OCCASIONAL CLASSROOM 
INSTRUCTOR OR VISITING VENDOR 

Janet R. Salas, Ph.D. 

ABSTRACT 

To ensure occasional instructors delivir 
instruction that 1s systematically organized Duke 
Power's Production Training Services has developed an 
Incidental Instructor Training Package in self-study 
and group workshop formats. This paper describes the 
training package, Its Implementation and results. 

INTRODUCTION 

At Duke Powerj there is a commitment to the Systematic Approach to 
Instruction. All full time classroom Instructors complete a two week, 
seventy-two-hour Instructor Training course which they must pass before 
they deliver any classroom instruction. With this program well esta- 
blishedj Production Training Services recently tackled another training 
need - that of the occasional or incidental instructor. Incidental 
instructors are defined as those plant subject matter experts, visiting 
vendors, or guest speakers called upon to deliver Infrequent or one-time- 
only training. It Is important that trainees In these occasional 
Instructors classrooms receive instruction that Is also systematically 
organized even though most of these Instructors probably will not have 
knowledge of the systematic approach or preparation of performance 
objectives. In addition, Production Training Services wanted to 
establish a method to verify ahead of time that any scheduled vendor 
training did meet training needs and followed the systematic approach to 
training. With budget constraints most companies face it is vital the 
vendors selected to provide Instruction are required to supply assurance 
that the training to be delivered does meet training needs, will be 
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effectively delivered, and achieve desired results. To resolve both of 
these instructional situations, Prodyctlon Training Services developed 
Incidental Instructor Training and a key organizational tool - The 
Incidental Instructor Preparation Plan. This paper will describe the 
training package developed, how it has been implemented and results to 
date . 

INCIDENTAL INSTRUCTOR TRAINING PACKAGE 

Two formats have been designed to serve our three nuclear stations 
individual needs- a self-study manual and a group working format. The 
training package also includes a facilitator's guide for conducting 
individual or group training, a video-tape, exercises to be completed by 
participants and evaluation forms to allow for participant feedback. 

THE INCIDENTAL INSTRUCTOR SELF-STUDY MANUAL 

The self-study manual is designed to train an individual who has 
his/her topic clearly In mind, 1n four to six hours followed by a 
de-briefing with a trained facilitator. The facilitator is close by to 
answer questions any time during training but 1s especially Important 
during the de-briefing when he/she evaluates the end product - The 
Incidental Instructor Preparation Plan - prepared by the participant - 
and offers clarification, guidance, and suggestions for Improvement. 

THE INCIDENTAL INSTRUCTOR GROUP WORKSHOP 

The Incidental Instructor Lesson Plan allows a trained facilitator 
to train eight (8) participants in approximately six (6) hours. Using 
interactive discussion, a videotape and exercises the facilitator guides 
participants through stages of learning, effective use of motivation, 
language, and questioning techniques. When all participants have 
completed an Incidental Instructor Preparation Plan each participant 
briefly gives an overview to the group of (1) how the Information or task 
will be taught, (2) training aids that will be used, (3) questions to be 
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asked and (4) points to be watched for to assess trainee^ s learning* The 
group gives feedback to the Incidental Instructor on clarity, logic of 
sequence, appropriateness of terminology, training aids, pace of 
training, and any crucial delivery problems that may be noted. The 
facilitator covers any content or presentation problems the group may 
have overlooked. At the end of the session, the facilitator collects the 
Incidental Instructor Preparation Plans for more thorough evaluation. 
The reviewed plans are returned in a day or so with more detailed 
comments that may have been missed or were not appropriate to share with 
the entire group. After each training session (Group Workshop 
or Self-Study) conducted facilitators return participant feedback evalu^ 
atlon sheets to Production Training Services. This information is 
assessed to determine how the training is being received and areas where 
it may need to be improved or revised. 

THE INCIDENTAL INSTRUCTOR PREPARATION PLAN 

This five page worksheet was developed to help an incidental 
instructor with no previous knowledge of effective systematic training 
prepare an organized classroom presentation. The plan brings to mind 
important items an ineKperienced Instructor needs to consider including 
(1) attention to purpose, (2) motivating introduction, (3) sequence of 
points to be discussed, (4) development of training aids, and (5) 
effective summary. To reinforce training a handout containing guidelines 
to review Just prior to teaching is also included to remind the 
occasional instructor of items to remember to do before the presentation, 
during the presentation and after the presentation. 

Attached to the plan is a suggested letter format which can be sent 
to vendors (or guest speakers). Items suggested to be covered in the 
letter include: 

• an outline of expectations of the vendor/guest speaker 

• a list of objectives the vendor/guest speaker needs to 

cover In his/her prisentation 
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available time for the presentation 

training material s/audio/vi sual equipment available for 

the vendor's use 
date, prior to training, vendor must submit objectives, 

lesson plan, materials for review, approval or 

possibl e revi si on 



FACILITATOR TRAINING 

Facilitator who deliver the Incidental Instructor training have 
completed a seventy-two (72) hour Instructor Training course or a similar 
course and a three (3) hour Incidental Instructor facilitator training 
program which trains participants to use the entire training package, 

INCIDENTAL INSTRUCTOR RESULTS 

The Incidental Instructor Training program was Implemented July 1, 
1986, To date, seven (7) facilitators and fifty-five (55) Duke Power 
occasional Instructors have been trained. The feedback from participants 
has been extremely positive. Incidental Instructors feel more organized, 
prepared, and confident. They particularly appreciate the guidance the 
facilitators offer. As one participant said, "I feel better about 
instructing a group now than I did before." 

Recently, facilitators met to talk about training effectiveness and 
the need for changes and additions. This feedback from facilitators plus 
comments from participants indicates a need to increase emphasis on 
delivery and practice. 

The Incidental Instructor Training Is off to a successful start. 
The next few years promise more efficient and effective occasional 
Instruction delivered by Duke employees as well as contacted vendors. 
The payoff will be in the results: 

1, Training which produces more competently trained, satisfied 
workers , 

2, The best use of our training dollars. 
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"BRINGING IT HOME" 
MAKING NUCLEAR POWER PLANT lR4INn^G 

RELEVANT 



Henry C. Billings 
Boston Edison Conipany 
Pilgrim Nuclear Power Station 
Plymouth, Massachusetts 



Brian K. Hajek, Stephen F. Puffcnbcrgcr 
Advent/Nuclear Multi-Image 

2691 Farmers Drive 
Columbus, Ohio 43085-2766 



ABSTRACT 

In 1985, a new series of multi-image training module^ was credited for the 
Pilgrim Nuclear Power Station^s General Employee Trniiiing (GETT) program. 
In the year since its initial deliveryj the multi-image coLiiponent of GET has 
been highly praised by both Boston Edison management ntid ENPO, Most 
importantly, the multi-image modules, along with A iiowly iniplementcd 
curriculum have increased the effectiveness of GET at Pilgrim/ 
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Hr^TRODUCTION 



... "^/'^^ SQluUoito this prablem is the use of videotape trainin.^ materials thit 

arc abundantly availabjcln the pas^, Pilgnm had bought and Lsedseveril of them h,^ 

ip^c up and improve , riling, wl^ilc kccjing the aucniion ^ cn^^y^ d 
ofleu.vely conimunicutins important information to new employees Mo i ™ 



MCULTI-IMAGE 



First, a word about llic medium Jcnown as Multi-Imace. Most nconlo think- nnt 
between). The projcgto^ i,,„y be con^centrated onto a single screen for a imximuin ^ 
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slide change speed nnd i^ingc coniplcxity (ns wc use at Pilgrim), or ihcy may be $p%ii(l 
out in a wiclc-scrccn fprimiat for a panoramic effect. It's even possible to control iSixiiii 
film, iighi^, or other special effects devices, in addition to the slide pixDjcctors, 

Miilli-Iiimgc hardw cire has made quantum, leaps in technology as computers ht^ve 
become mom sophisticated and less expensive. Current state-of-the-art equipment giycs 
the medium the sameninrfom access capabilities as videotape, and also makes it virtualls' 
impossible (barring niecli. nnical failure of the projectors) for the slides to go out of syiicS- 

But, wilh video so convenient to use, why did we decide to go with a multi-im^tp 
foiinat? The answers afc fourfold: 

1, Image qiiality: 

Our GET classes te^vcragc 30 people, in a comfortably large training room* 
adequate visibility by everyone, our image size is about 6-fect across. V/c fire 
curreiitly using oiir^-video projector wilh VHS tapes for certain modules, but ihcr^is 
absoliiicly no conH5=irison in quality. In fact, the BEST videotape machine can 
reproduce 525 lines from top to bottom of the screen, VHS or Beta machines ^re 
far worse, so if this is blov/n up to a big screen, the raster lines, color-shifts* (tirf 
misalignment froni a 3-iube video projector, become rather obvious. By coiitiia^f. 
the resolution of a typical slide measures over 4000 lines, and there is ncvcf any 
electronic distortidii_. 

High Rcsoluiion videotape fomiats are currently under development, but their 
intmdiiction is stiU years away. And, when they are inimduccd, high-rosoltUioii 
video projectors ^vill be cost prohibitive for most utilities. Even wl-cii 
high-resolution vidc^o is avaihible, it will still suffer from the other drawbacks of 
video, 

2. Flexibilily; 

With a videotape, v^hat's on the tape is on the tape. It cannot be changed. V/itHa 
properly designed I^^ulti-Image program, minor updates are easy , . . just shoofii 
new slide and drop it in! But not all multi-image shows arc obsolete-proof, (^^i 
producer was carcfisl to note which infoiination may be changed during the llfo of a 
show aiicl keep ilw specifics out of the soundtrack but presented on tho ^cr^^ii 
Then, when the specific item changes, a new slide can be dropped in . , . no spcciiil 
training necessnry, 

Sonielimcs, how^visr, audio changes are inevitable. With one of our prognin^s^!' 
change was nccessii in the soundtrack just weeks after it was delivered, Bcctiuso 
the sound and vi^tials are separate, it was very easy, and cost efficjent, for 
Advcnl/Nuclcar to s^c-do the soundtrack and bring us a new tape. The tupc cva 
changed the sequcndng of the slides for the edited portion, without having to jii^tc 
them all It was uneasy, convenient, no-compromise way to keep our show cuiWril. 
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3. Transferability: 



Even though a Muhi-Iniage show looks best projected on the screen it can also be 

ot olt-hitc GET classes, or, condiictuig small classus without tyin<^ up the ir^e 
classroom. Hosvevcr. :f any slides are changed, new transfers niii be mJula^ " 



4. Production Convenience: 



Photography in a nuclear power plant is a difficult undertaking. Bringing in a larce 
amount of equipment to adequately shoot videotape or filmis not only ^ fo dw 
security issue, but it also is a risk for the production crew as diSusa^Ss c5 dSS 
worth of equipment could be contaminated. And, lights and camSS caif cause -i 
mcuor interruption in work for employees. Photogra^y with MuhSnm^ SSie 
with a professional 35mm still camera and flash combination. Pho to^S^ C£Ui be 

- '^"^ t° '^'^ photographer Ir S^^pme^^ 

And, since a minimum of equipment is used, disruption of daily work is minimized. 

scmP J^n^P"?!?"'' ^^ si^^Prqicctor single^scrcen format was chosen to maximize the 
screen impact within the given equipment budget. The training room was almadv 
equipped wuh a rcar^projection booth and screen to which the system ada^d. ^ 

One of the benefits of today's Multi-Image systems is the avill-ibilitv nP 
equipment While Adven^uclear was very specif aliut our equ^,^ n^d?^ we 
weie able to purchase our system from a local AV dealer. The local Sler provides 

d^ lli'^S ."ff^f f^''^'^ ^'f "'°f ^"PJ^'y^'^fi backup units shouldluiySi^ ^ 
' ' » use, the only technical problems encountered have been nn 

occasional burned out projector lamp r.nd dirty tape heads. 



PILGRIM STATION GET PROGRAM SUMMARY 



In 1985, we had completed revising our curricukim for GET and we wished tn 

iS^pSsSi^ ourt,^" ' : - - that wo wanTcd mSia to 

Muhi liS^i^^^f = " ■ ' " ''-"^^ "^"='1 considcmtion, we chose 

MuUi-rmage. To fit o' ■ nvr,,, ,o took delivery of our first three modules in 



1) Industrial 6t. : . 

2) Radiation Proteclion Overview, and 

3) Boiling Water Reactor (Plant Familiarization). 
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In 1986, wc ordered three additional modules: 

1) Quality Assurance, 

2) Respiratory Protection, and 

3) Biological Effects of Radiation Exposure, 

(These last three have just been delivered after a prolonged production period caused by a 
strike that hit our plant last summer.) 

One of the very unusual ways chosen to treat the topics was to relate tasks uur 
workers perform at Pilgrim Station to tasks they perform at home. Some of the analogies 
entertain and use humor to teach. Others really make the subjects "click". Here's a short 
rundown on what we have: 

Theme: 

All the modules have one common theme running throughout: DO IT RIGHT!. 
This includes a common graphic treatment as well as a music string that fits. 
Advent/Nuclear discerned that this was the overriding message we wanted our 
workers to understand. If you're going to svork here, do it right! 

Boiling Water Reactor: 

For pai't of the "Plant Familiarization" segment of GET, a module on how a 
Boiling Water Reactor works was created. With eKtensive use of the slide 
animation capabilities of Multi-Image, we show neutron absorption, fission, the 
chain reaction, thermalization of neutrons, and water flow through a BWR, The 
amount of energy wc produce is related to numbers of coal cars, volumes of 
natural gas, and "six-million, seven hundred thousand 100-watt light bulbs." 

Industrial Safetyr 

A film on Industrial Safety has the tendency to be a "do this" and "do that" for 
things already well known. To improve on that approach, we chose to highlight 
three major safety themes: Safe Attitudes, Safe Conditions, and Safe Practices. 
Instead of continuously lecturing on these themes, several humorous scenarios 
are acted out, including someone tripping over an open file drawer while not 
paying attention, someone banging his head without a hiu'dhat, and someone 
coming to work with a hangover, thus encouraging trainees to be "mentally fit 
for work," 

Radiation Protection Overview: 

This was a real challenge. In 20-minutes we asked to have ALL the radiation 
safety instructional objectives normally taught in about 6 hours of classroom 
instruction capsulized. We wished to use it as a review of the regular 
instruction, just before giving the employees the test. Since delivery, we have 
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also found it invaluable for reqiialincation trainiim. It scivcs as an ideal 
retrcsher for people who luivc already had the training and who %vork with 
radiation everyday. 

Even with this usiuilly new topic, we make it relate to the worker's everyday 
life For instanee, the module begins with a coniparison of familiar radiations 
such as sound and sunshine, and relates how these can be found to be pleasant 
and how they can also be hamiful if their intensities arc too great. ' 

Quality Assurance.- 

Quality Assurance also posed a challenge, since it is a concept which can be 
difficult to communicate. We chose to meet this challenge by relating quality 
assurance to things people do everyday = from buying a car to fixing a home 
toilet. Tlien, the show cov;ers and summarizes the 18 criteria of Quality 
Assurance mandated by 10CFR50 Appendix B, and graphically shows how the 
criteria interrelate and are actually used in everyday life, even though svc don't 
quanufy and record that we have completed them. 

Respiratory Protection.- 

This module is designed for Level 2 workers who may use respiratory protection 
devices. It covers the reasons for using re.spiratory protection, and the exact 
procedures for checking out, inspecting, wearing, removing, and returning the 
lull face mask respirators used at Pilgnm Station. While not too many analo-ies 
arc used, the beginning highlights many of the warning signs of airborne 
coniamuuilion that workers need to be aware of ' 

Biological Effects: 

This modiilc is used to enliancc one of the eight Radiation Protection modules 
tauglu at Pi grim Station. It spends its 20 minutes relating the hcalih-physics 
concepts of biological effects in a very understandable manncn Included arc the 
energy of a milhrcm, acute effects, chronic effects, and the linear and threshold 
theories, all related to the risk of getting wet from exposure in a rainstorm The 
cluinccs of health effects from exposure arc compared to one's chances of 
winning the lottery, bnnging this fairly complex topic down to some simple 
common denominators. 



PRODUCTION TECHNIQUE 



Advent/Nuclear's approach to multi-imago production is unique. From our 
obKcrN'aiionsl^w mcclia producers use this technique to assure client saiisAiction from the 
hiaM. li s a Qualily Assurance program for media! 
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They begin by reviewing the plant's training objectives with the training staff - just 
what we want to teach in each module. In the ensuing diseussions, they make 
rcconimcndations relative to what objectives may be handled best with multi-image, 
other media, or lecture. 

Following this review and discussion with plant personnel, a Treatment is 
developed. The treatment is a glorified outline, describing just how they propose to 
"treat" the objectives selected, not only in the module In question, but also relative to the 
entire GET program. It includes pieces of script and ideas for visualization, transitions, 
and music. It's in this stage that everything is flexible. Without wasting a lot of time, we 
can see which ideas fly and w^hich ones don't. 

The Treatment is presented to the training staff for our approval. Our policy is to 
have the treatment reviesved by the appropriate subject matter experts in the plant Since 
it is only on paper, anything can be changed at this point, and usually, a lively discussion 
results in a revised treatment that fully meets everyone's expectations. 

From the Treatment a visualized Script is written, in the traditional fbmiat of 
visuals on the left and audio on the right. This is a collaborative effort between 
Advenl/Nuclcar's technical experts and script writers, and plant personnel, to assure all 
tlic audio and visual components will be authentic and correct. 

The script is then sent to plant subject matter experts for review and for approval. 
Since it has been written in such a cooperative manner, and based on the 
already-approved treatiiicnt, major changes are rarely encountered. 

From the script, the "Shot List" is prepared. This is sent to us, and we work to set 
up the appropriate shots, seheduling plant personnel to serve as models. A large number 
of plant personnel are used on screen to humanize the final product. The shooting 
schedule is finalized before pliotographcrs arrive on site. 

Outdoor scenes that relate the local plant environs to the program are also used, 
sometimes shot fi'om the air. Both utility line helicopters and private aircraft have been 
used for these shots. 

A number of scenes also are shot in Advcnt/Nuclear's Northwest Columbus studio 
or around Central Ohio. These arc generic scenes that do not require specific phint 
photography, but very effectively illustrate the objectives to be taught. 

Meanwhile, Advcnt/Nticlcar's artists work on creating the graphics for the show. 
These provide vividly colored animated graphics that simulate movement with still image 
technology. 

Also, the recording engineer receives the script and begins to lay down a rough 
soundtiiick in Advcnt/Nuclcar's recording studio. Music from their music library is 
selected, and often voices and sound effects recorded on site are added for complete 
authenticity. We are asked to approve music and voice selections. 
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Once the photographers return from the plant with photography, and the graphics 
and rough soundtrack arc coniplctcd, asscnibly bcguis. The sUdcs are cataloged, then 
sequenced with the script. Aj:omputcr is used to program the slide changes, using a 
technique similar to the SMPTE time code commonly used in video production. 

After the rough program is completed, a videotape "screen transfer" is made and 
sent to us, We can thus preview it at our convenience. We usually telephone our 
comments to the production team and they begin the "polishing" phase, where timing, 
image, and audio adjustm jii: ^ are made. 

After polishing is com; lete, the show is duplicated. The originals stay in 
Columbus, and the duplicate is prepared and delivered to the plant site. The 
Advent/Nuclear people take our trainins staff through the operation of the show, making 
sure we understand how to screen it successfully. We were also left with thorough 
written instructions. 

Advent/Nuclear keeps the originals for several reasons: First, if something happens 
as it is shipped to the plant, the original slides are not lost. Second, if something should 
happen to our slides, Advent/Nuclear can quickly duplicate a new set and send out the 
needed replacements, Thirdly, when the time comes for revisions, Advent/Nuclear 
revises the originals, and sends the updates to us, so we never have down-time while a 
show is revised. 



RESULTS 



The results of the Multi-Image training materials at Pilgrim Station have been very 
gratifying. The most noticeable effect is a significant increase in attention span from our 
trainees, thanks to the freshness of the media. Seeing themselves or other workers they 
know on the screen also contributes to a positive attitude toward the training sessions by 
the employees. 

The most recent INPO review of GET at Pilgrim praised the Multi-Image portions 
without criticism, and commented "(the Multi-Image) component is indicative of a strong 
commitment to general employee training in radiological protection." 

In summary, we are very satisfied with Multi-Image as a way of supplementing our 
training efforts. Advent/Nuclear's expertise and creativity have succeeded in helping us 
make our General Employee Training unique. 



PRESENTATION 



Our presentation will include a screening of several segments from the Pilgrim 
Station GET program to illustrate how we use everyday examples to relate plant 
requirements to the trainees. 
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PROPOSED COMBINATION OF TRAINING AND EDUCATION TO 



MEET THE BACHELOR OF SCIENCE REQUIREMENTS 



Albert E, Wilson 



ABSTRACT 



Tlia basic siTnilarities and differences of the education and 
training which ^ in the author's opinion , are actually needed by 
reactor operators are outlined and compared with the NRC require- 
Tnents^ Examples of engineering degree progranis are presented to 
demonstrate that they are NOT the appropriate educational goal for 
a senior reactor operator. A possible program of study which could 
be implemented Jointly by a utility and a nearby college or university 
is presented p The program combines both education and training to 
complete the requirements for a bachelors degree* Those student- 
operators entering the program should be able to work as auxiliary 
operators while pursuing the degree part time and qualify for the NRC 
Reactor Operator exam in five years. Then 5 while working as RO's, they 
should complete the degree requirements in another year* After an 
additional year of RO eKperlence, they should meet the NRC require-- 
ments for Senior Operator* 

Finally, some of the possible pitfalls of such a program are dis- 
cussed* These include Such things as drop-outs , union agreements , 
inflexibility of educational Institutions and, of course, cost. 



All of the utilities operating nuclear power plants have improved 
their training programs for plant personnel and most, if not all, of 
these training programs meet the standards of the Institute of Nuclear 
Power Operations (INPO) * However, none of them will meet the guide-- 
lines of the Accreditation Board for Engineering and Technology (ABET) 
for accreditation as engineering degree programs* The goals of TNPn 
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and ABET are vastly different and it would be virtually impossible to 
meet the accreditation requirements of both with a single program, ABET 
conaiders engineering to be a "design profession", as do most enginaerln| 
educators. The ABET criteria are not the same as those of INFO. A 
recent ABET publication^ stated i 

These criteria are intended to assure an adequate 
foundation In science^ the humanities and the social 
sciences, engineering sciences and engineering design 
methods, as well as preparation In a higher engineering 
specialization appropriate to the challenge presented 
by today's complex and difficult problems. 

If it were not for the mind set of some of the people In the Nuclear 
Regulatory Commission (NRC) that an engineering degree in the control 
room will solve all their problems, we could probably dismiss the 
engineering degree as inappropriate Just on the basis of the above 
accreditation criteria* 



If education is Important ^ and it surely is, then we should develop 
appropriate educational programs and convince the NRC that they are, 
indeed, appropriate. Also, If the educational program is equivalent 
to one leading to a bachelor of science degree , then a degree should 
certainly be awarded at the end of the program. One such possible pro- 
gram is presented at the end of this paper. 



PROGRAM IMPLEMENTATION 

In order to Implement a training/education program at a nuclear 
power plant which will lead to an acceptable bachelor of science degree, 
you will most certainly have to work closely with a nearby college or 
university. It will require some changes in attitude of the leaders at 
both the utility and the educational institution to make such a program 
work. For eKample, most educators don't believe that learning to do a 
job well is "education" while most utility managers probably don't 
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understand why the plant operator should have an appreciation of Tnusic . 
(The operators themselves probably don't undarstand that either,) 
However, knowing how to do a job well is important to the plant and 
having a broad education Is a requirement for the degree* 

It will help a great deal in working out a suitable program if you 
have someone at the plant who has had considerable experience in a 
college or university, preferably in an administrative capacity. You 
need someone who speaks the right language. For examples colleges gen- 
erally do not grant academic credit for "on-the-job training"* However , 

there are many programs ^ such as nursing and education ^ which require, 

2 

and grant credit for, a "practlcum"* Webster defines "practicum" as 
A course of study designed especially for preparation 
of teachers and clinicians that Involves the supervised 
practical application of previously studied theory. 

So make sure you treat your operators as "clinicians" and always 
say "practicum" and not "OJT" • 

The instructors which you use in the program are also going to have 
to be approved by the college, I don't know how it became established 
that doing some (usually useless) independent research and publishing 
the results in some (usually obsure) scientific journal was a necessary 
requirement for teaching courses which carry college credit but that is a 
"fact" which you will have to consider* 

The above examples may seem a bit facetious but , in all seriousness 
if you are asking an educational Institution to grant their degree for your 
program, your program has to, somehow, be made to fit their mold. 

THE PROGRAM 

The curriculum for one possible program is given in an AppendlK to 
this paper p I have called the degree "BS in the Physics of Reactor Oper- 
ation" but you must recognize that there are usually political implications 
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to any new degree. Early in the planning, a Task Force of leaders from 
both the Local State College (LSC) and the Nuclear Power Plant (NPP) 
should be forTned to work out potential problems, A few of the potential 
problems which this Task Force may need to resolve, in addition to the 
name of the degree, are given below* 

Curriculum 

The possible curriculum given in the Appendix is a very rough out- 
line of what might be appropriate* The cc ill have to be carefully 
screened by the Task Force or an academic r ^sory committee established 
for that purpose. New courses will almost cercalnly have to be approved 
by the Curriculum Committee of the LSC. 

Faculty 

The NPP will have to hire faculty who meet, or exceed, the minimum 
requirements of the LSC to teach the special courses. These persons ^ 
while employees of the utility, will require adjunct appointment at the 
LSC, 

Course Scheduling 

Scheduling of courses at the NPP presents a particular problem if the 
student^operators are working a rotating shift. Assuming a six crew 
rotation, it should be possible for all students to attend a class which 
is taught twice during the same day? once from 0800 to 1100 and again from 
1245 to 1545, The day crew and relief crew could trade off ^ the training 
week crew could attend either, the swing crew could come in early, and the 
mids crew could stay late. Anyone with that day off would have to come in 
for one session or the other. Two such courses, perhaps one on Tuesday 
and one on Thursday, would fit the schedule in the Appendix. Each course 
would meet 17 times for three credits. 
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Costs 

The program is going to be expensive. How you work out the details 
of possible compensation for back shift employees attending day classes 
and those coniing in on a day off will depend on your particular par-- 
sonnel policies. Also, any regular LSC faculty who teach at the plant 
should probably be given extra compensation unless the college counts it 
as part of their regular teaching load. There will also have to be extra 
operators in the programs at least six, to accommodate the semester on 
campus and 5 of course , the NPP must hire special faculty for part of the 
program. 

Retention in the Program 

Serious thought needs to be given to the question of retention. 
IiJhat happens when someone gets part way through the program and fails a 
course? How many times can he fail before he is dropped from the program? 
What happens if one quits the program? Questions such as these need to be 
answered by the Task Force before the program is implemented. 

Other 

There are a host of other questions which must be answered early in 
the planning process. A few of the questions which will face the Task 
Force are: 

How do you fit in the Requal program for operators? How does some^ 
one with previous experience *'test-out" of part of the program? \^at 
about vacations? How can work be made up if a student is sick? How 
does an operator get in his control room time during the campus semester? 
What transfer credit will be accepted? l^at if a regular college student 
wants to take one of the NPP courses? 

The specific answers to these and other questions will depend on the 
particular Nuclear Power Plant and Local State College. I don't feel that 
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any of the problams are insurmountable, however , provided the 
following three conditions are met: 

(1) The utility believes that it is in their best interest to have 
degreed operators and is ready and willing to pay the bill. 

(2) The Local State College sees the program as important to their 
mission of filling the educational needs of the region* (And they 
mi^fht make money from the program^) 

(3) The operators really want an education and a degree and are 
willing to work for it* 

CONCLUSIONS 

Maybe some of you have tried such a program. I know that several 
have similar cooperative programs set up at the associate degree level. 
I don't know of one at the bachelors degree level , I'm not even sure it 
can be done* However ^ if such a program can be implemented, I think it 
will lead to a higher knowledge level for your senior operators , increased 
job satisfaction for the operators in general , and more stable operating 
crews . 
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APPENDIX A 

POSSIBLE CURRICULUM FOR THE PHYSICS 
OF REACTOR OPERATION DEGREE 



YEAR ONE 
Fall (NPP) 

Present Aiixiliary Operator 
Training Program and initial 
qualifications. Should also 
include remedial math and 
other academlca as required* 

Summer (NNP) 

Analytical Geometry,,, 3 cr . 



Spring (LSC) 

First semester at LSC 
English Composition , , , . 
Speech 

Algebra and Trig ........ 

Chemistry , 

TOTAL . 



, .3 cr . 
. .3 cr. 
_5 cr- 
, ,5 cr, 
.16 cr. 



YEAR Tl^O 
Fall (NPP) 

Calculus I * 3 cr. 

Classical Physics 3 cr . 

(mechanics) 
Practicum for Turbin Bldg, 5 cr. 

TOTAL. ..... 11 cr . 



Spring (NPP) 

Calculus II. * , . , 3 cr. 

Classical Physics 3 cr. 

(electricity) 
Practicum for Reactor Bldg,... 5 cr. 

TOTAL. 11 cr, 



Summer (NNP) 

Classical Physics,,. 3 cr* 
(heat 5 light & sound) 



YEAR THREE 
Fall (NPP) 

Atomic and Nuclear Phys . , . 3 cr , 

Electricity, 3 cr . 

Practicum. 5 cr . 

TOTAL. . . , , , 11 cr . 



Spring (NPP) 

Nuclear Reactor Physics, 3 cr. 
Fluid Mechanics ............... 3 cr * 

Practicum 5 cr , 

TOTAL. 11 cr , 



Sumirier (NPP) 
Heat Transfer 



3 cr. 
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YEAR FOUR 



Fall (NPP) 

Reactor Dynamics- , , * , 
Radiation Chemistry.. 
Practicum. 

TOTAL , 



Spring (NPP) 

3 cr , Reactor Control 

3 cr. Radiation Protection..., 

5 cr. Practicum , , 

11 cr, TOTAL, 



3 cr, 

3 cr, 

5 cr. 

U cr. 



Summer (NPP) 

Mitigating Core 

Damage 3 cr. 



YEAR FIVE 
Fall (NPP) 

Control Room Time Practicum 5 cr . 
Training Reactor Week...,,, 1 cr, 
TOTAL 6 cr. 



Summer (NPP) 

NRC Reactor Operator 

Exam 0 cr. 



Spring (NPP) 

Control Room Time PracticuTn, . * 5 cr . 

Simulator Training. 3 cr, 

TOTAL 8 cr. 



YEAR SIX 
Fall (NPP) 

Computer Programming. 3 cr. 
Personnel Mgt, & 

Supervision 3 cr 

Practicum, 5 cr . 

TOTAL 11 cr. 



Spring (LSC) 

Second Semester at LSC to 
complete the general 

education requirements 15 cr. 

TOTAL, , , 15 cr , 
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ABSTRACT 

EMt. This proposed development effort would 
accomplish three major objectives^ as followsi 

1. To Identify and verify, through job analysis, 
the fl£it i g a 1 _ pr QfaaslQBAl^taaka that must be 
performed by electric utility instructors* 

2* To adapt and revise existing instructoir 

tr P i nl^a^Miilaa to make them self-contained 
and highly specific to the professional 
knowledge and skills needed by electric utility 
instructors. 

3. To develop n^w in^trugtor _ training iBQguleEF if 
needed, to meet utility instructor training 
needs that are not addressed by any eKisting 
materials. 

Smi. It is anticipated that approximately twenty 
(20) modules will be needed to address all of the crlti- 
cal instructor tasks identified during the job analysis 
phase* The National Center for Research in Vocational 
Education proposes that it would be very cost-effective 
and time^ef f icient to cooperatively undertake the devel-- 
opment of the needed instructor training modules with a 
consortium of about to ten interested electric utility 
companies. 
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On November 1^ 1986^ six electric power coiTipanies 
and the National Center for Research in Vocational 
Education at The Ohio State University organized the 
Electric Utility Instructor Training Consortium. The 
companies giving leadership to and sponsoring Phase I of 
the Consortium includedi Cleveland Electric^ Detroit 
Edison, Duke Power, South Carolina Electric and Gas, 
Tennessee Valley Authority, and Virginia Pov^er, 

The purpose of the Consortium is to achieve three 
major objectives designed to develop and upgrade the 
professional skills of new and continuing instructors 
within the ira3ustry. The three objectives are as 
follows I 

m Identify the critical professional tasks that 
must be performed by electric utility instructors 
by using job analysis 

# Adapt and revise eKisting instructor training 
modules to make them highly specific to the 
professional skills and knowledge needed by 
electric utility instructors 

• Develop new instructor training modules to meet 
training needs that are not addressed in existing 
materials 

WHY A CONSORTIUM 

High-quality employee training programs are essential to 
the safe, reliable, and efficient production of electric 
power* However, such training is only as effective as 
the instructor conducting it. "Training the trainer" is 
therefore an activity essential to any company's employ^ 
ee training efforts. 
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The Electric Utility Instructor Training Consortium 
provides a cost-effective rnethod for preparing training 
materials that support the development and the impLove'- 
ment of the skills of nev/ and continuing instructors. 
Sharing the cost of materials development through a 
consortium allows individual companies to receive great- 
er benefits than they could afford working alone. 

HOW DOES THE CONSORTIUM OPERATE 

The consortium is directed by a board represented 
by one member per company- Upon joining and financifLlly 
supporting the consortium^ each company has an equal 
voice in setting policy and providing input into work 
activities. Staff from member companies serve as tech- 
nical consultants in the development of inaterialsr and 
also review all materials to ensure their accuracy and 
relevance to the electric utility work environment. 

The Consortium staff at the National Center manage 
the v7ork activities and are responsible for conducting 
job analysiSir task analysis^ task verification,, and 
module conceptualization^ developments revision^ and 
publication. 

Funding to support Consortium activities comes from 
membership fees paid to the National Center. The fees 
support work authorized by terms of the membership con- 
tract and in accordance with the policies established by 
the Board of Members. 
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The current Consortium scope of work consists of 
two phases. Phase I, the initial job and task analysis 
task verification, and task clustering for module devel 
opment, took place during the period November 1, 19 86— 
February 28, 1987. Phase II, involving the materials 
development, field review, revision, and publication 
began March 1, 1987, and will continue through June of 
1988, at which time, 20 modules will have been devel- 
oped. 



Work on Phase I began in earnest on November 12-15, 
1986, when eleven representatives of the six member 
companies convened at the National Center in Columbus 
for the Electric Utility Instructor DACUM Workshop. For 
those of you who may be unfamiliar with DACUM 
(Developing h Curriculum), it Is simply a structured, 
small group, modified brainstorming type of process for 
conducting a very hlgh-guallty job analysis of any 
occupation in a short period of time at low cost. The 
National Center has used the DACUM process extensively 
since 1980 in its own curriculum development efforts and 
has conducted over 100 workshops for various industry, 
business, governmental, and educational agencies. 



The DACUM committee was selected by asking each 
member company to provide two of their most expert in- 
structors. To provide fair representation of all as- 
pects of instruction, the Consortium Board specified 
that the experts were to be Identified by stratifying 
the committee around fiv specialities i (1) classroom 
instructor, (2) shop/lab instructor, (3) OJT instructor, 
(4) simulator instructor, and (5) Instructional develop- 
ment specialist. At least two persons were selected to 
represent each of these specialities. 
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Under the leadership of two National Center experi-- 
enced DACUM facilitators^ the committee worked hard for 
four days to identify all of the duties and tasks they 
believed to be important to the instructor's job. 
Before reaching closure on the fourth day/ the committee 
was asked to review three literature-basGd task lists " 
to help insure that no important tasks were overlooked. 



A few tasks were added as a result of this 
procedure but mostly the committee felt reassured that 
they had indeed done a thorough and high-quality job. 
When closure was reached/ the committee had identified 
130 tasks clustered into 12 duty areas* The duty areas 
include i 

A. Develop and Maintain Technical Proficiency 
B* Develop and Maintain Instructional Proficiency 
Assess Training Needs 

D. Develop/Revise Instructional Material 

E, Prepare for Instruction 

P* Coordinate and Schedule Training 

G* Operate and Maintain Instruction Equipment 

H, Deliver Instruction 

I. Supervise Trainees 
J* Evaluate Trainees 

K* Evaluate Training Effectiveness 
L. Perform Administrative Activities 



The next task was to develop a verification instru-- 
ment for submitting the tasks identified by the DACUM 
committee by mail to five to ten instructors in 13 
electric power companies. The instrument was submitted 
to all member companies and to several other companies 



1 The literature^based task lists Included the Region I 
generic instructor task list assembled in 1986, the 
INPO Instructor Job Survey compiled in 1986, and the 
Pennsylvania Power and Light task list for simulator 
instructor developed by Janice Reitmeyer. 
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who indicated an interest in Phase II and a willingness 
to participate in the verification procedure* 



The verification respondents were asked to rate 
each task statement on (1) the importance of the taskr 
(2) task learning difficulty^ and (3) frequency with 
which the task is likely to be performed^ using a six- 
point Likert scale ranging from O^-S, h total of 120 
instructors f employed by 13 electric power companies and 
who worked at 19 different plants^ responded to the 
survey by the initial cutoff date. 



In addition to the questions asked about each taskr 
questions were also asked about such items asi 

(1) Number of instructors employed by the company 

(2) Number of years served as an Instructor 

(3) Occupational area of assignment 

(4) Type of instructor training received 

(5) Adequacy of the training received 

(6) Worker traits and attitudes most important to 
being a successful instructor 

{7) Type of training materials most valuable for 

new instructors 
(8) References found to be most valuable in their 



A report entitled fiiimmaxE^^^lafik-^jLif iMtifln 
Cat ax^^l 9 82 ji££txi£_Ut±lityJnsi£iafl wa s 

prepared and submitted to member companies and to the 
Institute of Nuclear Power Operations (INPO)* 



As of this writing, the 130 tasks have been veri=- 
fled as important and have been tentatively clustered 
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for module development purposes. A significant number 
of additional electric power companies has also shown 
considerable interest in possibly joining the Consortium 
for the Phase II development effort, A report on the 
current status of membership will be provided at the 
time this activity is reported on at the Orlando April 
1987 meeting* 



BENEFITS OF THE CONSORTIUM 



The benefits of developing instructional materials 
cooperatively are manyi 

m Each company receives 20 modules specific to its 
instructor training needs, yet pays the cost for 
only 2 modules, 

• By sharing development expertise, the National 
Center and member companies produce higher quali^ 
ty materials than could be produced alone, 

• Use of the resultingf lndustry"-specif ic modules 
will help companies produce more qualified in= 
structorsr which leads to more qualified employ- 
ees . 

• Companies will be better able to meet 
accredlation standards for industry instructor 
training programs* 

• Company personnel participating in the module 
development process will grow professionally v;hen 
they discuss common problems and solutions with 
their counterparts from other companies* 



WHAT IS THE NATIONAL CENTER 



Since many of you are unfamiliar with your National 
Center for Research in Vocational Education, I hope 
you'll permit me to describe the organization and its 
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purposes briefly. The National Center is a full- 
service, non-profit, research, development, training, 
and information systems organization located at The Ohio 
State university. in operation since 1965, the organi- 
zation was officially designated the National Center for 
Research in Vocational Education in 1978 by the U.S. 
Department of Education based on a successful but very 
competitive bidding process. 



The National Center's mission is to increase the 
ability of diverse agencies, institutions, and organiza- 
tions to solve educational problems relating to individ- 
ual career planning, preparation, and progression. The 
National Center fulfills its mission by; 

• Generating knowledge through research 

• Developing educational programs and products 

• Evaluating individual program needs and outcomes 

• Providing information for national planning and 
policy ^ 

© Installing educational programs and products 

• Operating information systems and services 

• Conducting leadership development and trainina 
programs ^ 

The National Center ensures high-quality service In 
this effort by combining its over 20 years of experience 
in developing performance-based currlculuin for teachers 
and educational administrators with its 10 years of con- 
sortia leadership. Six consortia with over 70 members 
presently operate under National Center leadership to 
provide a range of services to members. The organiza- 
tion is in a unique position to be able to develop high- 
guality performance based materiala for the electric 
utility industry because it: 

• Has had extensive and very successful experience 
m developing 132 performance-based teacher 
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education (PBTE) modules designed specifically 
for the preservice and inservice education of 
vocational teachers , 

Has had extensive and very successful eKperience 
in working with 14 state departments of education 
since 1978 through a consortium arrangement to 
develop 34 competency-based modules and 11 guides 
for vocational administrators , 

Is responsible for the National Center Clearing-- 
housef as well as the ERIC Clearinghouse on 
Adult/ Careerf and Vocational Education, 

Operates the National Academy for Vocational 
Education^ which has conducted over 600 workshops 
and conferences on a wide range of topics* 

Employs a highly professional staff that is 
experienced in many fields^ Including business^ 
industry, and labor; and in the development of 
performance-based training materials. 

Has successfully managed over 900 contracts 
including over 25 contracts with various compa- 
nies in its 21 years of existence. 

Has the world's largest library of materials on 
all phases of career development/ preparation^ 
and advancement. 

Is an integral part of The Ohio State University 
and can readily access its extensive human and 
material resources , 
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JOB RELEVANCE OF ENGINEERING AND SPECIALIZED EDUCATIONAL 
PROGRAMS FOR LICENSED REACTOR OPERATORS ~ 

Barbara D, Melber 
Lise M, Saarl 



ABSTRACT 

This study investigates the extent to which traditional 
baccalaureate engineering degree programs and specialized 
educational programs for reactor operators cover academic 
knowledge needed for licensed operator Job functions. Academic 
knowledge items identified by a job analysis were systematically 
compared to the curricula of a sample of the educational 
programs. ApproKlmately 65% of the academic knowledge 
identified as necessary for the positions of RO and SRO is 
taught in college level engineering courses. College 
engineering curriculum provides considerable material beyond 
that identified as necessary for licensed operators. There is a 
great deal of variation among specialized programs for reactor 
operators^ ranging from coverage of 15% to 65% of Job-related 
academic knowledge. Half of the schools cover at least 50%, and 
half cover less than 30% of this knowledge contents 



INTRODUCTION 



This paper analyzes the job relevance of (1) accredited baccalaureate 

engineering degree programs, and (2) specialised college credit 

educational programs designed for licensed nuclear reactor operators. 

12 

The purpose of these studies * , carried out for the U,S* Nuclear 
Regulatory Comralssionj was to determine the extent to which existing 
educational programs covered academic material necessary for carrying out 
the job functions of nuclear reactor operators. 

Academic knowledge identified by a job analysis prepared for tha 
Institute of Nuclear Power Operations (INPQ) as necessary for reactor 
operator functions was systematically compared with the content of 
curricula of baccalaureate degree engineering programs and with tailored 
educational programs for reactor operators. The INPO job analysis 
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identlfted over 200 knowledge categories (e.g., division) across 12 major 
aubject areas: Mathematics, Electrical Science, Chemistry, Materials, 
Classical Physics. Nuclear Physics, Reactor Theory, Instrumentation and 
Control, -leactor Plant Protection, Health Physics, Heat Transfer and 
Fluid Flow, and Engineering Drawing. 

Subject matter axperta (SMEs) in engineering curriculum and nuclear 
operations training carried out the comparison task to identify the 
coverage of Job-related academic knowledge by the • -ious programs. 
Eight engineering schools offering B.S. degrees in mechanical, 
electrical, chemical, and nuclear engineering and eight schools offering 
tailored educational programa for reactor operators Including non-degree. 
A.S. degree, and B.S. degree programs made up the sample of programs 
studied . 

For the traditional engineering B.S. degree programs, the SMEs 
indicated whether or not a particular academic knowledge item was 
covered, and If covered, ac what levels (1) a high school level 
prerequisite to entrance, (2) in the first two years of the core 
curriculum, or (3) in required upper division major courses. For the 
specialized educational programs the SMEs Indicated whether or not each 
knowledge Item was (1) a prerequisite to the program. (2) covered by the 
program curriculum. 

Extent of Job-Relevant Knowledge Covered by 
B.S, Engineering Degree Programs 



Overall, there was considerable consistency across the SMEs in ratln 
the items on the knowledge list. Only 5% of knowledge categories could 
not be asslBncd specific educational level ratings due to a high degree 
of variation in ratings among the SMEs. Thus, it appears there is 
substantial similarity across engineering schools in the coverage of 
Items on the knowledge list. 
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The distribution of the educational level ratings for the over 200 
knowledge categories on the total knowledge list Is shown in Table 1. 
For example, 9% of the knowledge categories were rated as being covered 
in high school courses required for entry into college engineering 
programs ♦ 



Table 1* Distribution of Educational Level Ratings for 
Total Knowledge List 



Educational Percentage 
^^^Q^ Distribution 

High School 9^ 
Core Curriculum 17% 
Upper Division 47^ 
Nuclear (36%) 
Other than Nuclear (11%) 
(Chemicals Electrical ^ 
Mechanical) 



Not Covered 



22% 



No Rating^ 5^ 



^Due to insufficient agreement among SMEs as to whether knowledge 
category was covered or not covered. 

Table 1 Indicates where the knowledge is taught by specific 
educational levels and majors. The percentages Indicate the percerit of 
knowledge categories covered in required courses. It is thus a 
conservative estimate of coverage of Job-related knowledge because 
elective courses are not included. The percentages do not indicate what 
an individual student would cover because students major only in one area 
and take a considerable number of elective courses in addition to 
requirements. This table shows the extent to which Job-related academic 
knowledge is college^level material, 

A major conclusion of Table 1 is that of the academic knowledge 
required for nuclear power plant operators, about 65% is covered in 
college engineering required courses. To translate In terms of 
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Individual students, it can be said that a student with a bachelor's 

degree in nuclear engineering would cover, at a minimum . 62% of the 
academic Job-related knowledge (9% In high school prerequisites, 17% in 
core requirements, and 361 In nuclear engineering upper division 
requirements) , 

These findings indicate that much of the academic knowledge needed by 
nuclear power plant licensed operators Is covered in college nuclear 
engineering curriculum. Thus, it might seem reasonable to consider a 
college enginaerlng degree as an educational qualification. However, it 
is also important to consider whether a potential educational 
qualification may be excessive. In other words, a large proportion of 
the academic knowledge needed by nuclear power plant licensed operators 
could be obtained by earning a baccalaureate degree In nuclear 
engineering, but there may be much more academic content In a nuclear 
engineering degree program than what is required on the job for a nuclear 
power plant operator position. 

The SMEs estimated the amount the covered Job-related academic 
knowledge represented of the total material covered in the engineering 
curriculum. There was a great deal of variation among the SMEs In these 
estimates, Indicating both the difficulty of making such a global 
Judgment and variation across individual school curricula. However, the 
mean ratings indicate that from 60% to 80% of the ftnglneering core 
curriculum covers material outside that which was specifically identified 
as Job-relevant by the Job analysis, and from 40% to 60% of the nuclear 
engineering upper division curriculum covers material not on the academic 
knowledge list for reactor operators. 

Ixtent of job-Relevant Knowledge Covered by 
Specialized Educational Programs 

The sample of eight programs studied cover the spectrum of the types 
of specialized educational programs balng used by the nuclear Industry 
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for licensed reactor operators* All of the programs In the sample serve 
workers at nuclear power plants who are currently licensed or in training 
for licensing at the RO and/or SRO level(s)i two of the programs are 
directed at Shift Technical Advisors, In addition, some of the programs 
serve a general student population in preparation for entry into reactor 
operator positions in the nuclear industry. In most caseSj the 
continuing education section of the educational institution coordinates 
the development of programs, which generally involves combining courses 
from engineering or engineering technology departments, science and 
mathematics departments, and at times junior college courses and plant 
training courses which have been evaluated for granting of college 
credits . 

The three coursework programs are all associated with degree 
programs s so that students may apply credits earned toward a degree. Two 
of the coursework programs serve Shift Technical Advisors as well as 
Senior Operators. The three associate degree programs are all in nuclear 
technologys in contrast to the two bachelor degree programs in the sample 
which are not in nuclear technology nor in nuclear engineering* 

The three coursework programs (Schools C-1, and C^3) ^re drawn 

from the wide range of these typ^ of programs. School C-3 encompasses 36 
credit hours, compared to 66 hours for School C=l and 122 hours for 
School C-2 (converted to quarter credit hours). All of the coursework 
programs are comprised of a set of required technical courses | they do 
not have elective or non^technlcal courses as part of the program. 

The A,S, degree programs (Schools AS^l, AS-2, and AS^3) also vary in 

terms of total credit hoursj (from 65 to 120 credit hours). All the A,S. 

degree programs also require some credits in non^technlcal areas such as 
huraanltles and social sciences. 

The B.S, degree programs (Schools BS=1 and BS-2) are the most 
extensive in terms of total credit hours (203 and 19.2 liours, 
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respectively). Their number of required technical credits are similar to 
the more extensive coursework and A.S. degree prograras, but they have the 
greatest non-technical credit requirements. They also Include technical 
electives as part of the program, which generally is not part of the 
coursework and A.S. degree programs. 

The extent of coverage of the complete list of job-related academic 
knowledge by specialized educational programs was determined by computing 
the percentage of the over 200 knowledge categories from the academic 
knowledge list receiving a mean rating of "substantial coverage" Cmean 
score greater than 3.5 on a 5 point rating scale). When at least two of 
the three raters for an individual program indicated the content of a 
knowledge category was not directly taught in a school curriculum but 

prerequisite to other content in the curriculum, the category 
was designated as prerequisite. Where this prerequisite knowledge would 
be covered was not Addressed ; presumably it would be covered either in 
high school level coursework or In plant training courses taken prior to 
entry into the specialized educational programs. 

The results indicate that there is very little knowledge list content 
that is prerequisite to the content taught in the specialized educational 
programs. On average, only 4X of the knowledge categories were 
identified as prerequisites^ the individual programs ranged from 2% to 8X 
of the knowledge list content as prerequisite material. 

A review of the overall coverage of identified Job-related knowledge 
by individual programs Indicates that the curriculum of School C-2 
provides the greatest coverage of the knowledga list content; 63% of all 
the knowledge categories are taught in this coursework program. School 
AS-1, School BS-2, and School C-1 all cover approximately half of the 
total knowledge list content. The remaining programs, Schools AS-2 and 
AS-3, School BS-1, and School C-3 cover less than one third of the 
knowledge categories. In five of the eight schools (C-l, C-2, C-3, AS-1, 
and AS-2) approximately two thirds of the technical coursework covers 
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Items from Che knowledge list; in the remaining three schools 
BS-l, BS^2) a little less than half of all the raqulred technical 
material is devoted to the knowledge list items. 

The most striking feature of the school profiles is the variation 
across the individual programs i The schools differ substantially in the 
extent of Job«related content being taught. 

Combining uiha results of the individual schools by program type 
indicates no differences among the types of programs in terms of extent 
of overall coverage. On the average ^ each type of program covers 
approximately 35%=40% of the knowledge categories. This is due to the 
variation across individual programs within each typej described above* 
Two of the three coursework programs ^ one of the three A,S, degree 
programs, and one of the two B,S. degree programs cover half or more of 
the total knowledge list, while one coursework program^ two A,S. degree 
programs, and one B,S, degree program covers less than a third of the 
knowledge list content. 

These findings demonstrate that there is no particular type of 
specialized educational program that consistently provides more 
substantial coverage of the job-related knowledge than other program 
types • 

Comparison of Subject Matter Coverage by Specialized 
Programs and B,S, Degree in Nuclear Engineering 

Mathematics was the only subject area where a considerable portion of 
the job-relevant knowledge was a prerequisite to the math courses in each 
program. For most of the programs, between 1/3 and 1/2 of the 
mathematics knowledge categories were prerequisites to the mathematics 
taught in a curriculum. Combining prerequisites and direct coverage in 
the curriculum. Mathematics also was the subject most cumpletely covered 
across all schools. At completion of the programs , students would have 
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covered at least 2/3 of the MathemAtles knowledge on the knowledie list 
In seven of the eight achools. 

Atomic and Nuclear Physics is substantially covered by six of the 
schools, ranging from almoat 60^ to over 901 coverage of knowledge 
categories in that subject area. In ftridltlon, five of the schools cover 
the majority of the knowledge categr rles In the areas of Reactor Theory 
and Health Physics, 

Reactor Plant Protection is not covered at all by five schools and is 
taught mlniraally by two schools Cwith lass than 25% coverage of this 
area) , 

Chemistry and Engineering Drawing are the subject areas least covered 
In the specialized educational programs. Engineering Drawing Is not 
covered by any of the programs. Seven of the eight schools cover little 
or none of the Job-related chemistry Identified on the knowledge list. 
Four teach none; three teach leas than 20%. This is likely to be due to 
the narrow focus of cheraiatry that la directly relevant to nuclear 
reactor operators, which Is limited primarily to water chemistry. This 
specialized area of chemlstty Is only one small area covered In basic 
coilege-level chemistry courses. 

The individual schools vary with re=pect to coverase of the remaining 
subject areas. For example, half of the schools cover most of Heat 
Transfer (from 66% to 88%), while half cover little or none. The 
curricula of the individual schools In the areas of Electrical Science, 
Materials, Classical Physics, and Instrumentation and Control, similarly, 
are spread across the range from ralnlnial to substantial coverage of the 
knowledge categories within these subjects. 

Turning to the Individual programs. School C-2 provides substantial 
coverage of most of the subject areas with the exception of Chemistry and 
Engineering Drawing. Schools AS-1, C-1 and BS-2 also cover a broad range 
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of the subjact areas. School AS-3 appears to be focused primarily in the 
areas related to physical covering some Job^ralated knowledga in 
Mathematicsj Classical Physics, Atomic and Nuclear Physics, Reactor 
Theory, Health Physics, and Heat Transfer, and not covering knowledge 
identified in the other subject areas* School C-3 teaches courses 
primarily In the areas of Materials, Atomic and Nuclear Physics and 
Health Physics. Schools BS-1 and AS-2 cover some material across a 
broader range of subject areas, but only two to threa of the subject 
areas show substantial coverage. 

The variation across the schools most likely reflects the tailored 
nature of these specialized educational programs. These A*S. degree, 
B,S. degrea and coursework programs are, for the most part, used to teach 
specific areas in conjunction witb particular plant training programs. 

The nuclear engineering B.S. degree programs cover both basic 
fundamentals (Mathematics, Electrical Science, Materials, Classical 
Physics, Heat Transfar and Fluid Flow) and nuclear-'oriented subjects 
(Atomic and Nuclear Physics, Reactor Theory, and Health Physics) quite 
extensively. Coverage of these subject areas (Including prerequisites) 
ranges from 60% (Electrical Science and Heat Transfer) to 100% (Classical 
Physics and Atomic and Nuclear Physics). 

The B.S.N.E, programs are similar to the specialized programs Jri the 
vary limited coverage of Job^relatad chemistry and engineering drawing. 
They differ from the specialized programs in their lack of coverage of 
Instrumentation and Control, 

CONCLUSIONS 

The major conclusions of the comparative analysis are i 

0 ApproKlmately 10% of the academic knowledge needed by ROs and SROs 
is covered in high school courses that are prerequisites for 



290 



IV, 3 JO 



entrance into college engineering degree programs. This knowledge 
is primarily in the subject area of mathematics. 

ft A substantial amount (approximately 2/3) of job^related academic 
knowledge Is covered In college engineering courses* 

- Apprcicimately 65% of the academic knowledge identified as 
necessary for the positions of RO and SRO is taught in college 
level engineering courses, either In the core curriculum taken 
In the first two years (17%), in upper .ilvislon requirements for 
the nuclear engineering major (36%), or In other upper division 
majors (IIX), 

- Approximately 20% of the academic knowledge needed by ROs and 
SROs Is not covered either In high school or college engineering 
programs. This material is primarily in the areas of I water 
chemistry, reactor plant protection, and engineering drawing. 

College engineering curriculum provides considerable material 
beyond that identified as necessary for licensed operators, 

- Overall, no more than half of the total engineering core 
curriculum and the total upper division nuclear engineering 
major overlaps with academic knowledge needed by licensed 
operators. 

There is a great deal of variation among specialized programs for 
reactor operators, ranging from coverage of 151 to 65% of 
Job-related academic knowledge- Half of the schools cover at 
least 50%, and half cover less than 30% of this knowledge content. 

There Is no systematic difference in the Job-relatedness of the 
different types of specialised educational programs f A,S, degree, 
B.S* degree, and coursework. 
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• Traditional B,S, dagree programs in nuclear enginesring covar at 
laast as much job^relaCed knowledga as most of the apeclallied 
educational programs* 
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i3XPERIENCES IN SOLVING THE CHALLENQES OF ON-SITE DEGREE PROGRAM S 

J, M» Chris tenson L. E. Eckart 

Nuelaar Ei?^ :^>iearlng Program 
Unlvarglty of Cincinnati 

The Uolvaralty of Clnelnnfttl <UC) Nuelear Engltiearlng Program 
Faeulty has now had six years of experience in dellvarlng ©n-slte 
educational programs to nuclear power plant technical personnels 
Programs of this type present a variety of challenges to the faculty, 
the management of tha client utility and to the students who become 
involvt.d in a partieular program* This paper describes ho^ each of 
these groups can Identify and suceesstully solve these ehallenges. The 
solutions we describe are drawn from our own experiences whieh have beei 
described in some detail elsewherel^3* Other solutions to these 
ohallenges are eertainly possible* We make no claim for the particular 
ones wa offers beyond the fact that they have worked over a sustained 
period of time and that results they have produced have left all three 
parties mutually satlsfleds 

FACULTY CHALLENGES 

The first and foremost challanga to a nuclear anglnearlng faculty 
is deciding whether on^-stte education is an activity that It wants to 
undertake* The U.C^ Nuclear Engineering Faculty Is relatively small (6 
full-time members) ayid supports a full range of ©n-campus academic 
programs at the BS, MS and Pm levels. Faculty members are also 
involved £n funded research and consulting activities. Supporting an 
on-site (and off-campus) educational program is obviously an additional 
commitment by the faculty and one that will Inavltably subtract from the 
faculty man-hours available for on-campus activities. Is the extra 
effort worthwhile? What are the baneflts to the Nuclear Englnaering 
Program as a whole and to tha faculty mambers as individuals? Are these 
benefits worth the effort and risks that such a non- trad! tlonal program 
entails? Obviously the answer by our faculty has bean in the 
affirmative^ but In our view it Is essential that tha faculty seriously 
consider tha pros and cons of becoming involved in an on-site program 
before it undertakes such a i-ommltmant. The outcome of such faculty 
considerations has two distinct benefltst 

IV. 8,4.1 
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I. If the outcome is a detislon to proceed with an on-slta program, 

then th.re is a high probability of widespread faculty consensus and 
wllUngness Co contribute to the program. Our exparlence Is that a 
worthwhile on-site proiram involves so much effort that the 
existence of the foregoing conditions ar. essential prerequisites to 
a successful program, 
2, The considerations produce a thought- provoking debate and analysis 
of the options and trade-offs involved in different types of on-site 
program. The outcome helps to define the type of program that the 
facuty regards as being academically appropriate and worthy of their 
Involvement. 

At UC we have gone through this process on at least three 
occasions. Each time the consensus faculty view has been that 
Involvement in an on..slte program is sufficiently positive that the 
effort should be undertaken provided that the program can be properly 
structured, the faculty effort level can be controlled, appropriately 
compensated, and planned for in an orderly fashion. Realizing that 
these featurea are essential preconditions to faculty support and 
Involvement, we have bee« able to shape our progr.nm proposals so that 
they have had these characteristics. Having a clearly defined se. j£ 
necessary conditions for faculty acceptance prior to entering into 
Internal and external negotiating sessions has allowed us to leave such 
sessions with client and University administration approval of the 
desired policies. It is unlikely that such an outcome could have been 
achieved without the prior existence of a consensus faculty viewpoint. 

UTILITY MANAGEMENT CHAI,L1NGES 
The management of the client utility sponsoring an on-site 
educational program must answer questions similar to those considered by 
the faculty. Is such a program worthwhile? Does it produce sufficient 
benefits to justify the resources It will consume? Again these 
questions are most appropriately considered prior to making the declslor. 
to sponsor the program. In all the programs we have been associated 
with there Is no doubt that the management has seriously debated these 
questions, although not always on the time scale we suggest. 
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Generally we have not bean participants In the utility management 
sessJLons where these ^^uestlona havf»^ been considered. All we can report 
are the answers ti*e sessions have produced as reflected in the policy 
positions stated by management representatives* Certalrily the primary 
jus tlf lea tion for the utility management of the first U.C. ©n*-site 
programs was the perceived necessity of responding to the TMI Action 
Plan Requlyoments set forth by the NRG In late 19£0*^'*5 In that era the 
simplest answer to the foregoing questions for utility management was 
"mandated by regulatory requirements". Even at the tlmei however , some 
elements in utility management reallEed that there were other reasons 
i:or providing on-site educational programs for their technical 
personnel^ and one of the U.C. programs was requested and sponsored for 
non-mandated reasons* Currently our pereeptlon is that although 
regulator mandates still play a role In the utility decision-making 
process I they are no longer the primary reason for utility sponsorship 
of our programs. 

The primary benefits from the utility management viewpoint appear 
to us to be the following I 

1, Some degree of genuine enhancement in the technical proficiency of 
on-site engineering personnel. 

2. A reasonably objective , externa 1 measure of thcj intellectual 
abllityj profession?! maturity and depth of professional commitment 
of a portion of their engineering staff* The usefulnass of 
objective Information of this type (even though incomplete and 
imperfect) can be of significant benefit in making personnel 
decisions. 

3* A safety-valve for the more determined and ambitious on-site 

professional staff. The rate of turnover among the professional 
staff is a continual and legitimate concern of the utility 
management. The perception of staff members that management Is 
providing a way for them to develop and advance their professional 
skills furnishes some degree of relief from the common dally demands 
of their Job assignments* The significance of this aspect to 
utility management is enhanced because the Individuals who become 
involved In on-site programs are often exactly the same people who 
are most important in maintaining the utility's technical 
proficiency on an Independent basis. 

' O Q e 
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4. The availability of an on-site program m«k« slta employment 
significantly movm attractive. This feature can contribute 
aignlflcantly In the recruitment of new employees and In fflaklng site 
assignments more attractive to currant employees. 

STUDINT CHALLENQIS 
In our experience the prlma^ challenge facing a student who 
becomes Involved in an on-site program is finding the time to make the 
extra effort that academic courses require while still carrying out 
their dally professional responsibilities. Since these responsibilities 
often seem to require more than 40 hours per week and at least 
occasional traveling, it is a real challenge for a student to arrange 
his schedule so that he can simultaneously undertake academic course 
work. A further challenge is provided by the necessarily "long-haul" 
aspect of programs that meet all of the operational constraints and 
still have the desired characteristicsi On-site programs (at least for 
professional engineering personnel who usually have many other essential 
commltmenta), are always part-time and often after-hour efforts for the 
students. With these constraints it is Inevitable that a multi-year 
commitment Is required on the part of the student if he Is to obtain a 
degree. 

ONE SET OF SOLUTIONS I THE PEP PROGRAM 

U.C. with the support of Cleveland Electric Illumnlatlng Company 
(CEI) has developed an on-site graduate level educational program (PEP) 
for professional engineers at the Perry site. The program Is now in its 
third year of operation and provides CEI employee, with the opportunity 
to earn a Masters degree In Nuclear Engineering. CEI (and other on-site 
engineers) can also take selected Individual courses for purposes of 
professional development without pursuing the entire M.S. degree 
program. A unique feature of the program Is the use of the "mixed-mode" 
format to deliver most of the Instruction. The mixed-mode format 
appears to be the optimum way to deliver a multi-year graduate program 
to an off-campus site, particularly when all of the instructors involved 
have a variety of other commitments. 
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PEP was Initiated lii the S^.mmer of 1984 after Bmvmt^l maetlnga 
batwean GEI managainent Mdfteumty members of U^C^'a NucU^r EnglnMrlng 
frogram* Thm outcome of tliiaf ^saetings was the daelaion — ^ proeesd with 
th€ program and to provide It th the folLowlng features i 
1* Thm majority (at least thra^-quartirs) of tha inattucttton to taka 

place at the Wmvvy alte, 
2, The delivery tmtm of thi «owTse matirial to be at a fapold enough 

paoa that parttcipants caii thay ari making real pr»graas towards 

a graduate degrae* Oni ftou&se par calandar quarter tha delivery 

rata deeldad upon. 
3* Tha course daltv^ry mu to Be compatibU with tha cQQ^traint that 

tha partcipants have ^O^hoy^ per week profassional ©blL .gationa whila 

a course is batng taught* 
4* Tha coursa delivery wodi ta liava suffleUnt flaKlbility to 

accomodata tha naads of part iclpants who miss an oGeasi ^^nal class 

beeausa of shift work isilgn^raants of higher priority i;^T~of asalonal 

obligations, 

5* An intagral part of tlii ptog=3ram will be a practically o^^ianted 
thesis or project (piebably ^Farry-speclfic) carriad out by tha 
participants on an iyidivielui^l baslsi 
6, All program costs In.cwid tha Univariity of Ctncinnsitl to ba met 

by Clavaland Elacttic, 
Savaral mathods of ceyrsi d^s livery ware InvastlgaCsd t^^fore a 
mijcad-moda dallva^ foi^'raatvas decided upon* In the mlKed^'^Kode delivery 
format, coursas ara prmgantid in a combination of In-'petsofi lacturas and 
videotapa films. Typically, thf^a of the four couraaa aff^t— each yaar 
use the mixed-mode ^^.^at. The Seurth eourie is delivered an-person by 
a faculty mambar who spettds eue «^y each wiek on^slta. In el^ thar format, 
the class meats for % 1/2 hoyra^ ona evening par weak for a total of 10 
waaks# Studants also have hfintt?^*^^k and study assignments w^Klch 
typically require an addlciflnal 5^ — 7 hours per waek. 

In tha mlxad^modft forniiti tii^m Instruetor conducts tha f "±rst class 
sasslon in-parson whloh BUm hi^m to astabllsh parsonsl C0a^_tact with 
the students, distribi^te books, p Tcocass rigistratlon materia Is, presant 
the first lecture and aKpLaln tha course foifinat. The lnstru»^tor can 
also use this visit to eoLliet tti^hnlcal information about tX^a Perry 
Plant that can ba uaad mm %mpl%^m for the eourse. 
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The aecond week typically beglnf the first of three, 3-week cycles 
of videotape insttucttoifa and In-peraen lectures. Each cycle commences 
with b#o weeks of vldeot=*pe dellviry folia-,,ad by a third week in which 
the videotapes are 8uppU^B,ented with an liL_-p.r8on lecture session. The 
last week of the third 6:ycle culmlnawa liu_ a 1-hour in-person review 
session foUowed by a fl»al eKamlnitlon conducted by the Instructor. 
During the videotape delivery modi M,,eks, the Instructor Is available at 
a specified time (telephone of flee hours) -to receive telephone calls 
from members of the claas, eithti Indivtclti^lly initiated, or on a 
conference call basis. 

The selected dellvfl>icy mode has the advantage of being able to 
reschedule videotapes fc= partialpants who have not been able to be 
present for the schfldulea showing, It also has the advantage that an 
entire cours,, can be rep«.ated at a later d»te at a considerably reduced 
cost. Despite the many aeniands on their tttaie 65 students have 
partlpated in at least OT^m or mott of the Elrst eleven PEP courses. The 
first M.S. degree traduataes of tha prograa are projected for June 1988. 
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GETTING THE MOST OUT OF TR^Itqi^jG^Nj ^ EPUC^T — lO^ 
THROUGH PERSONAL ^W MSS 

John E. Carroll 

It should be a fairly simple prsmise to say chat fehe sum total of 
all of the efforts in Education and Training should bg a safe, efficient 
and productive operation. 1 would narrotg the focus ay^n more for the 
moment, however , to the point of concantradoii on Che Safety aspect. 
Hans Blix, the Director General of the International M romic Energy Agency, 
in his presentation at the INFO GEO Workshop in Novemba 1986 addressed 
this when he pointed out that, '^Safety, whether in avla ^tion, neuclear 
power or any other industry^ is a cont inuQUsly evolving ; business," He 
continued, "The occurrence of two significant neuclear ^ accideni s in two 
leading neuclear states in the course of aeven years ha^ m raised questions 
about whether, and to what extent, some biilc safety pr- Inciples might 
usefully be defined to which all States Vsuld more sttOKigly commit them-- 
selves * " 

Now an initial reaction could rightly be that we hMve always had 
safety of operation as the foundacion of all planninf f«^r our Education 
and Training programs. In addition, our aoreening and testing of job 
applicants, the state-of-the-art of training equiptnent «nd the automation 
of various functions and safety equipment have all baeti calculated to 
arrive at an absolute minimum of incidenta or accidentg^ With all the 
attention that has been given in this regard^ i^hy than Lis it a fact that 
in 1983 and 1984 more than 50% of all Sigxilficant Bvencte in the U. S. 
Neuclear Industry were a result of inadequata Human Performance? Why is 
Che record unchanged today? 

In the report on the Chernobyl accident, the IncerrL^at ional Atomic 
Energy Agency stated, "Complacency by operators, manager and infrastruc- 
ture^ engendered by many years of successful operation a ^nd by the belief 
that the unexpected cannot happen, appears to have beeti . a root cause of 
the Chernobyl accident. It was a major contributor to cr^e Three Mile 
accident, and it is a central issue in the management of neuclear safety 
at any reactor." Other root causes in this a.id other Significant Events 
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have been the human incerface with hardware, procedures and, to my mind 
most importantly 5 other humans. 

The traditional approach in Education and Training has been to pri- 
marily address the human interface with hardware and procedures and we 
can cake pride in knowing the effort has been successful when we recog= 
nize that today the causal factor in the majority of our events is no 
longer a problem with the hardware or the procedures. Specifically, at 
Chernobyl, again quoting Hans Blix, "We learned that flagrant disregard 
by operating personnel of safety rules and procedures during a testing 
operation placed Chernobyl -ynit 4 in an unstable state from which it 
surged out of control." 

Whenever an investigation into one of these incidents or accidents 
is conducted, the focus seems -o be to try to determine what happened. 
I would suggest we should shift the emphasis more to wh^ it happened. 
Wh^ should individuals who have been through an intense selection, educa-- 
tion and training process perform in less than an optimum manner? Espe- 
cially when the results could be catastrophic? If the answer can be 
found, then the approach to formulating Education and Training programs 
could be refocused to provide more satisfying and safer results. However, 
rather than looking for the answer to "why" individuals have acted the way 
they have in these events, our reaction, in most cases, has been to look 
for ways to design the individual out of the system - to automate more 
and thereby make the system more inherently safe and forgiving of operator 
errors. This approach can be carried just so far, though, before we have 
to deal with the underlying problem of human action and Interaction. 

There could be the temptation, on the part of some Utilities, to say 
that since they haven't had any incidents or accidents of this type that 
it is not their problem. Consider, however, that an^ event that receives 
public attention impacts the entire industry and not just the utility in- 
volved. In his address, Hans Blix made the point that "INFO neuclear 
safety programmes were developed after TMI, out of recognition that the 
collective fortunes of the members depended on the fortune of each indi-^ 
vidual member and that the credibility of the strongest depends In no 
11 measure on the credibility of the weakest," At the same CEO work- 
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shop, Bill Lee, Chairman of Che Board of Duke Power Company, followad 
Hans Bllx on the program and put it even more plainly when he said, ^*Each 
of you is hostage to the performance of the poorest in this room!" 

I know that every utility has addressed the subject of Human Factors 
and that courses for supervisors and managers have included subjects to 
enhance their abilities. The programs have covered items such as Leader- 
ship, Interpersonal Communication^ Command-responsibilities and Limits, 
Motivation of Personnel, Problem Analysis and Decision Making. In most 
cases, though, there is no opportunity for the individual to become aware 
of how their implementation of the knowledge gained in these areas impacts 
those with whom they interface and therefore, how the overall operation is 
affected . 

Two recent publications, Good Practice TQ 503 and NU REG/CR-425S have 
addressed the subject of Team Training. The emphasis in both cases is on 
individuals gaining the recognition, the personal awareness, of how they 
operate in concert with others and that synergy can only be achieved 
through a team effort. It should also be noted that every member of a 
team needs to be involved in the training and not just the supervisors. 

The aviation industry's accident experience closely parallels that 
of the utility industry and they have embarked upon a program of training 
their cockpit crewf^ that I think can be considered as easily transferable 
to the utilities control room personnel and ultimately to the plant posi-- 
tions with whom they closely relate. 

There are many versions of Team Training or Resource Management 
Training, but one of the more thorough and proven programs is one imple- 
mented by United Airlines in 1981. The goals of the program were, in 
part, to establish an atmosphere of openness in the cockpit, to allow 
each individual to gain an insight into their style of operation, to 
understand the various comniand and leadership responsibilities, to under-- 
stand how external influences can affect the operation and how synergy can 
be achieved. 

To reach these goals,, it was agreed that the training had to be 
given to all crewmembers, that it had to be a participative program^ not 
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a passive one and that it had to be made a part of the recurrenc trainin| 
as well. 

A precept of any team training should be that it allows for the rec= 
ognition and subsequent internalization of the need for a behavioral 
change^ if necessary ^ on the part of the individual* United *s course 
enables this to take places because the traditional approach of a stand- 
up form of instruction is not usedp but rather the learning takes place 
primarily in a team environment ^;here one's peers supply the feedback 
from which the individual benefits. 

The first phase is the use of a text to establish the frame of ref^ 
erance to be used. Next, a seminar is scheduled in which the intellectua 
understanding of team, dynamics is applied in a time-^constrained atmospher 
Recurrent training is then given each year^ this time in the simulator, 
using scenarios designed to emphasize the need for teamwork. This train-- 
ing Ll video=taped and selected portions played back in the debriefing 
period for peer discussion rather than the traditional instructor critiqu 

In the six years since the training was implemented by United^ there 
have been some very gratifying results. Broad acceptance by the crewmem-- 
bers is evidenced by the reports written by Instructors and Supervisors* 
Individual crews have stated how the principles and concepts imparted in 
the training have enabled them to handle irregular and emergency situa^ 
tions more effectively. Accident investigations have shown how use of th 
results of the training have minimized the seriousness of a situation* 
The most important result ^ however ^ is the recognition that individuals 
have gained on how they impact the operation and that what counts is What 
is right, not Who is right. 

Before the inception of the Resource Management training. United *s 
experience in the loss of airplanes was only slightly better than others 
in the industry, both domestically and worldwide. Today United 's record 
is four times better than the rest of the world. While it may not be the 
only factor^ Resource Management Training, or team training, certainly 
has had an unquestioned influence. 

Anyone embarking on a team training program must know what the 
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objectives are and then answer the following questions: 
^ What should be the content of the training? 

How should the training be given? 
^ When should the training be given? 
^ Who should conduct the training? 

If a thorough approach Co the planning and implementation of this 
form of training is taken, the desired results can be achieved. If not, 
it will be "jusc another training program-- , a square filler that wastes 
time^ energy and resources. Worse yet, the Human Performance causal 
factor will still dominate in our Significant Events. 

United has extended the principles of their team training into every 
facet of their programs for cockpit crewmembers. As a result, it has be^ 
coma apparent that every training experience is more productive than had 
been true in the past. With this recognition , I feel secure in saying 
that a heightened personal awareness gained through team training will 
enable every individual to get the moat from their future education and 
training experiences- 
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TEAM TRAINING PROCESS 
FOR ' 

NUCLEAR POWER PLANT TECHNICIANS 



A,C, Macris 



INTRODUCTION 

The purpose of team training is the cooperative and 
coordinated actions of individuals to attain a common goal. 
Such training requires the development of more sophisti- 
cated educational techniques than those previously established 
for training individuals alone. Extensive research has been 
conducted to devise methuds and techniques to bring about 
effective team training* This paper discusses current team 
training methods and presents an instructional strategy for 
the application of effective team training techniques. 

CURRENT TEAM TRAINING 
Only recently have attempts been made to define train— 
ing systems that result in proficient teams. The difficulties 
with present-day team training are categorized and explained 
in the following sections* 

Individual Training in a Group 

Current "text training" is primarily individual train- 
ing in a team context* Instructors administer programs 
that focus on the application of individual skills. 
Individual skills and knowledge are necessary prerequisites 
to team training* Training of individual skills is fairly 
straightforward/ but the techniques used to train "team 
behaviors" are quite different. Team actions are based on 
communication and interaction within a structured group of 
individuals* V^^am training will not be effective when the 
individual skills are not on a par with those of other 
members* A team cannot develop team behaviors when team 
members are being used to compensate for a lack of any 
team member's individual proficiency. 
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Training Through Testing Techniques 

Current team training typically begins with a short 
classroom session in which pertinent aspects of the team 
task are reviewed. This is followed by a series of exercises 
designed to provide individuals with experience in their 
respective jobs. 

The typical instructor's role is one of recording his 
observations for a critique of the exercise results. The 
team is later gathered for discussion of errors. This 
approach to training stresses the negative aspects of per^ 
formance while providing little positive feedback for correct 
and desirable actions. As a result/ teams acquire a good 
deal of practice in a trial-and^error fashion. 

The most desirable approach to learning is a combination 
of correction of errors and reward for proper goal-directed 
behaviors. This shaping procedure is designed to promote a 
positive and successful team training environment. 

Research has established that correction must be based 
on feedback that is properly timed and appropriate to the 
level of the team* Many factors must be considered when pro- 
viding knowledge of performance to a team: 

o The information to be provided. 

o When to provide the information. 

o The form of the information feedback. 

o Who provides the feedback. 

o How much information should be provided at each 
stage of learning. 

Research suggests that individuals rely on feedback 
inherent in the tasks as well as information external to the 
immediate environment. This makes the source and timing of 
external feedback critical. Too much knowledge of performance 
early in training can cause the team to emphasize deficient 
areas to the exclusion of other important elements of a 
problem. 
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Defining Team Behaviors 

A variety of definitions exist for team behaviors. 
These definitions tend to confuse the establishment of atan^ 
dards for effective training. The most familiar team 
behaviors defined thus far ar municationa and coordination* 

Attempts at training these team behaviors have tended to 
revert back to individual training techniqueSp More precise 
definitions of team behaviors as related to training of 
technicians a^re provided in this paper which better define 
the team training needs. 

Team Concept 

Historically/ the structure and function of individuals 
designated as a team has not been well understood- In 
addition/ there have been inconstancies/ in definitions on 
what constitutes a team* This presents obstacles to the 
development of effective team training techniques. 

In summary/ difficulties with team training/ such as 
training individuals in groups/ attempting to train through 
testing/ inadequate definition of team behaviors/ and an 
inconsistent concept of team structure/ have contributed to 
the difficulties in applying the team training research in 
real world situations. 

APPROACH TO TEAM TRAINING 

Three steps are necessary in developing team training: 
o Team definition. 

o Identifying desired and appropriate qualities, 
o Development of necessary training techniques* 

The above represent the logical sequence for any training 
approach/ starting with statements of how is trained and 
ending with how the training is conducted. The last step in 
this approach is discussed in the "Instructional Delivery 
System" discussed later in this paper. 
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Team Characteristics 

Reviews of current ceam training literature provide 
diverse definitions of teams. Klaus and Glasser (1968) have 
determined a list of characteristics that are particularly 
well suited to operational team definition. These character- 
istics, with minor modifications, embody the concept of a 
team as it applies to the nuclear industry, 

A team is: 

o Relatively rigid in structure, organization and 

communication, 
o Well defined for member assignments so that each 

member<s contribution can be anticipated, 
o Dependent on cooperative, coordinated participation 

of several specialized individuals whose activities 

contain little overlap and who must perform their 

tasks to some minimum level of proficiency, 
o Often involved with systems or tasks requiring 

perceptual -motor activities , 
Q Able to be guided in on-the-job performance based 

on a task analysis of the team's equipment/ goals/ 

or situations. 

Definitions of Maintenance Teams 

The focus of team definitions has been directed at 
operational teams. These teams are relatively easy to define 
based upon the above characteristics and their defined 
structure. Definitions of maintenance teams is difficult, 
but can be generalized into three categories; Gore, Augmented 
and Support Discipline Teams. Each team is discussed below. 
These definitions are linked to the definitions of mainte- 
nance baselines discussed later in this paper. The size 
and composition of maintenance teams may vary depending upon 
the maintenance discipline and task. As a result, the 
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following presents definitions based upon these unique 
considerations . 

Core Team 

The core maintenance team is defined as, the requisite 
group of technical personnel within a specific maintenance 
discipline/ necessary to perform a specified task. In 
general/ the individual skills and knowledge of the team 
members is at a nominal level for frequent and routine tasks* 

Augmented Team 

The augmented maintenance tfsam is defined as the core 
team with additional members added from the same discipline* 
These members usually possess a higher degree of skill 
and/or knowledge necessary to complete a specific task* 

Support Discipline Team 

A support discipline tc-am may be a core team or an 
augmented team from another discipline. This team is intro-^ 
duced when task performance requires additional support in 
an area outside the expertise of the designated core team. 

Qualities of a Team 

In an applied setting/ team behaviors fall into two 
major categories^ interactive and attitudinal* Interactive 
behaviors deal with how team members relate to one another 
and how these interactions impact the accomplishment of team 
goals* Attitudes are reflections of member views of the 
team as a capable functional unit* 

Classical training system development starts with 
specific behaviors and translates these into skills/knowledge 
and resultant training objectives. Team training assumes 
that prerequisite individual skills and knowledge have been 
attained and focuses on interactive and attitudinal behaviors* 
Team behaviors are specified in Table 1. 
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TABLE 1, QUALITIES OF TEAM DEVELOPMENT 
Level I Level 11 Level HI 

Uses plant data to develop Iffectively uses most data Anticipates data needs and correlate 
useful information available all data available 

Agreefflent on team pro- Team proceduris Shifts to different procedures with- 

accomplished smoothly out disruption as tltuaclon changes 

Adapt to most situational Adapt standards to all Innovative response to emergent 

requiremencs situational requlrementfl requireaents 

Awareness of taam and goals Awareness of other mamber's Mutual support in overload condition 

needs and his individual 
requirements 

Cooperative Enthusiastic ^gresslve 

Produce required results Ready for new challenies Meet all situational demands 

WlUlni to supply team Maxlnize team effort Aware of team's dependence on him 

needs 

Understands role in team Good role understanding Ixhlbits camaraderie 

and team identity 
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Levels of proficiancy for these behaviors have also 
been defined. (See Table 1,) Level 1 is a formative stage 
in which individuals are assigned positions and given 
an initial goal to function as a cohesive unit. The result 
of this level is a competent functioning unit able to 
perform routine maintenance tasks. In Level II training/ 
the Individuals have as their goal a growth in knowledge 
and refinement of skills so that upper levels of team quali-- 
ties may be achieved. Intermediate level goals are projected 
as a logical midway training point between the extremes of 
having a complete absence of team behaviors and having an 
expert team. Level Ill's goal is to have an expert quality 
team capable of handling a wide range of emergent situations. 

To illustrate this progression in training/ consider 
"awareness". As an interactive behavior/ members of Level I 
must understand that they are part of a team with specific 
goals. One level further in team development (Level II) 
would be an awareness of other individual member functions 
and needs in addition to the awareness of a team situation 
and goals. An expert team (Level III) consists of members 
who mutually support one another in overload conditions/ 
which require an understanding based on the first two 
levels of team awareness. 

TECHNIQUES FOR TRAINING 

Separation of Learning and Testing 

Training sessions must be divided into processes of 
acquiring knowledge (learning) and testing of that knowledge. 
The Optimum environment for learning provides a moderate 
amount of stress to enhance motivation. Constant evaluation 
through testing inhibits learning and provides a minimum of 
learnable information to team members. Tests should logically 
follow the learning and acquisition stage of instrucf -Dn* 
The majority of the team training course is designed £or 
instruction. Evaluation is used to verify that the team has, 
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in fact, mastered specific objectives or to demonstrate 
training system effectiveness. Only after the team has 
demonstrated a readiness for evaluation should it be tested. 
Tests should positively reinforce achievement and monitor 
progress . 

Positive Approach to Training 

Learning theory and research have well established the 
need for constant positive feedback along with correction as 
the optimum learning environment. Positive learning techniques 
are aimed at preventing mistakes before they occur and thereby 
minimizing the need for negative feedback. The approach is 
guided by these points: 

o All training is designed to keep the team in control 
of the problem. Correct performance is rewarded by 
the instructor during training and afterwards during 
discussion of the training event, 
o Team training assumes that individual skills and 

knowledgas are compatible with the level of instruc- 
tion. When necessary, refresher training is included 
as a pretsam training condition, 
o Learning should occur under non-evaluation conditions. 
To ensure compliance with these conditions, the 
instructor gives at least a short introduction to 
training problems sensitizing the team to important 
exercise parameters, 
o Team performance diagnostics are separate from train- 
ing sequences, 
o The team is given opportunities to demonstrate 
achievement periodically/ usually at the end of a 
training level sequence. This assessment through 
demonstration can be a positive means of tracking the 
team's performance. 



o Teams have direct involvement in the overall learning 
process* Instructors encourage teams to self -evaluste 
their performance at the completion of an exercise. 

Integrated Approach to Training 

Implementing the positive approach to training is 
accomplished by integrating the overall instructional strategy 
with the design of instructional materials* Desirable team 
behaviors are developed in two fundamental phases. These 
phases will be referred to as Prerequisite Behavioral Aware^ 
ness and Maintenance Baseline Operations, The integrated 
approach comes about through an instructional delivery system 
that provides step-by-step instructor guidance. The instructor 
has leeway in adjusting the rate of progress to the needs of 
the team/ by accelerating fast learners or giving additional 
training for slower learners. 

Prerequisite Behavioral Awareness 

This first phase of the integrated approach attempts to 
ensure that uniform behaviors and attitudes ( charaGterist ics 
of productive teams) shaped and encouraged through group 
interactions. These interactions are designed to foster 
cooperation and mutual respect among group/team members. 
Achieving the characteristics of productive teams entails 
changing or eliminating behaviors and reinforcing positive 
behaviors through training. This in turn creates the funda- 
mental prerequisite training which is exercised, expanded and 
improved in the next phase, Maintenance Baseline Operations. 

Maintenance Baseline Operations 

Training logically begins at the initial competence 
level of the team. The word "baseline" means the "beginning 
at the beginning" concept, A baseline of a maintenance evolu-* 
tion is defined as the simplest form of the evolution repre- 
senting responses to normally encountered situations within 
optimum conditions * 
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Each baseline is a conceptual category of maintenance 
operations based on established goals. Although listed dis^ 
creetly/ baselines overlap and are eKpected to frequently 
co^occur. Coincident baselines are used in complex situations 
as adaptations to emergent conditions. 

The training of a team, therefore/ logically begins with 
the baseline alone. Once the baseline is mastered/ complicate- 
ing factors are introduced as a means of tempering teams to 
stressful conditions. Emphasis is consistently directed 
toward maintaining baseline procedures when accommodating to 
increased situational demands. Complications are" designed 
to be assimilated into baseline workings so that fundamental 
procedures and team actions remain relatively fixed. Adapta- 
tion to emergent conditions does not necessitate completely 
novel sequences of behavior from a team. Teams are trained 
so that baseline procedures attain a degree of automatici ty / 
leaving more time and mental energy for addressing the 
unpredictable. 

Training baseline behaviors and variations requires a 
formally structured training program. Each step is based on 
a clear objective for progressing the team from baseline 
performance to the expert level. 

B^our fundamental maintenance baselines have been defined 
in figures 4-1 thru 4-4. The specifics of each are dependent 
upon the evolution and maintenance discipline, 

TRAINING LEVELS 

Need for Levels of Training 

Teams must start training at levels appropriate to their 
capabilities and progress as far as possible during the 
designated training time period. Improvement with training/ 
depends on the particular characteristics of each unit (team). 

Teams should not be administered training beyond their 
skills and abilities. This can be demoralizing and confusing. 
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ABNORMAL MAINTENANCE BASELINE 



mRE FOR A TASK 



START A TASK 




MTE^TO-COMPLE]^ INFREQUiiiTLY PEBFORMED TASM 

TEAM AUGMENTED WITH TASK SPECim EXPERT 

TASK PROCEEDS AS PLANNiD REQUIRING COORDINATION 
WITH OTHER TEAMS 

TASK PROCEEDS ESSENTIALLY AS PUNNED XOQRDINATING 
TEAMS REQUIRE ADDITIONAL SUPPORT 

TASK BECOMES COMPUCATEDMDITIONAL TASK SPECIFIC 
SUPPORT IS REQUIREDJN ADDITION TO COOmiNATlON 
WITH OTHER TEAMS 



PERFORM 



fspm 
COMPLETE A TASK 
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Capabilities of teams vary according to combined past 
individual training/ experience^ and time spent together 
as a unit. The training level concept is a recognition of 
team differences in proficiency and an attempt to tailor the 
training system to each team's needs. Graduated difficulty 
of training sessions allows definite starting points tailored 
to a team's abilities and serves to identify training gains 
within and between levels * 

Three Training Levels 

Levels of team proficiency are difficult to establish 
absolutely and must be somewhat flexible* Proficiency of 
a team can be expected to vary somewhat from one task to 
another. Team capabilities are/ however/ expected to 
transfer across specific operations/ allowing general agree- 
ment about broad categories (levels) of ability. 

The first step in operationally defining categories of 
team abilities was the specification of team qualities 
(Table 1). The next step is the definition, in appropriate 
terms/ of what is expected from the team at each training 
level. The three training levels are defined as followsi 

Level I represents the minimum level of performance 
required of a core team and/or augmented core team/ such 
that the team can successfully perform the normal maintenance 
baselines. The goal of training in Level I is to have the 
the individual members of the team function as a unit. 
The team must meet the minimum requirements for accomplishing 
all normal baaalines with sufficient team behaviors and 
proficiency. 

Level II . A Level II team should function as a unit 
with increased proficiency/ capable of completing normal and 
abnormal baselines and variations. Completion of Level II 
training is designed to be the norm. Initially/ the norm 
will be arbitrarily set at a level based on Level II training 
objectives and subjective capability definitions. After 
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training is in places data collection over many team training 
periods will be used to identify specific performance stan- 
dards that can be normalized to the average. 

Level III represents advanced team achievement. This 
level represents complex interactions of baselines and 
variations where the team must exercise the most sophisticated 
of emergent qualities, Where Level III training is adminis^ 
tared/ it actually enhances the team's experience levels 
to an "expert" team* 

INSTRUCTIONAL DELIVERY SYSTEM 

Instructional delivery includes the training process, 
training techniques/ and the development of curriculum 
materials. Incorporated within these elements are the 
approaches to team training previously discussed. Team train- 
ing consists of behavioral awareness, refresher of behavioral 
aspects/ theory/systems refresher, operational refresher 
(as necessary), and team training. Theory and operator 
refresher training should be limited to those areas that if 
not refreshed could degrade team learning. The need for 
behavioral or, individual technician refresher training can 
be determined by instructors. Each team member has to be 
capable at a level compatible with Level I demands to 
effectively interact as a team member. 

Individual capabilities are expected to progress as 
the team progresses with experience and knowledge gained 
from time spent in the training environment. Through 
corrective feedback from instructors and other team members/ 
individuals become better technicians and better team members. 

The proposed training techniques have been designed to 
positively reinforce desirable correct behaviors within teams. 
The curriculum structure ensures that the team is challenged 
and interested but not overwhelmed. 

Team attitudes, a critical component of team motivation, 
are enhanced in this manner, resulting in positive feelings 
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about L ndividual and team abilities. Along with implementing 
this pc^oposed training strategy/ some form of instructor 
orienta^tion to the prograrn is necessary, it is difficult for 
instriia tors trained in traaitional techniques to quickly 
assimtl ate "new and improved" programs. Skepticism and 
trepida tion about a new p^ograni can be alleviated by pilot 
familia ^ization programs, A pilot program would yield infor- 
mation on the success and feasibility of the new training 
techniq o^a , 

The Training Process 

Te^m training is divided into sections/ each covering 
a major area of performance. Each section consists of topics 
related to training that address areas of performance, A 
demonat^ation exercise is used only after a team has mastered 
the con-tent of each section , Demonstration affords the team 
an opportunity to show its expertise for a type of task 
before proceeding to the next topic or section. 

Team Tr«.ining Exercises 

Fouar distinct componerits comprise each exerciser 
(1) preexercise diacussiort / (2) preexercise briefing, (3) 
administration of the exe^rcise/ and (4) postexercise discus-- 
sion* ^hese are essential, for each training exercise to 
foster positive attitudes toward learning and to reduce the 
probability of errors during exercises, 

o Preexercise discus sions - clarify the objectives/ 
intent/ and major elements of an exercise, 

o Preexercise brief i ngs - outline initial conditions 
of the ensuing exerccise for the team, 

o Administration of exercise ^ the instructors note 

team and individua i compliance with discussed goals. 

o Postexercise discu ssion - instructor leads the team 
in a discussion of the good and bad aspects of its 
performance. The instructor's attitude is crucial 
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at this point since his comments can color what the 
team thinks of its performance. Instructors can ask 
leading questions, offer suggestions about alternative 
actions/ and point out deficiencies or mistakes in 
a manner that constructively examines exercise results. 
Instructors are the pivotal factor upon which program 
success depends. The best training program design is useless 
unless implemented in the manner for which it was intended. 

Assessment Exercises 

Opportunities for teams to convince themselves of their 
abilities is important. An option for the instructor involves 
the use of assessment exercised. These exercises must, how- 
ever, cover only those elements previously trained, therefore, 
well within the team's ability. Upon completion of the 
assessment exercise, instructors can positively reinforce 
the desired team behaviors while making an assessment of 
performance. The aim of assessment is to show positive 
progress with a fair examination. 

Use of the Training Process 

The proposed training effort takes more instructional 
effort than traditional trial-and-error training, but also 
results in more efficient training. Increased efficiency 
saves training time since fewer exercises are necessary in 
this structured program. Using the baseline/variation-to- 
baseline structure, teams can develop the capacity to handle 
emergent situations with flexibility. An optimum predeter- 
mined learning sequence is followed to develop those team 
qualities shown in Table 1. Periodic assessment ensures 
progress and provides guidance to the instructor. 
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EVALUATION OF TEAM SKILLS 



FOR CONTROL ROOM CHEWS 



C. D, Geddy 
J* L, Koontz 



ABST^CT 

Although team training has received 
considerable attention throughout Industryi a 
systematiG approach to team skills training has 
only recently been proposed for control room 
craws. One important step of the approach to team 
skills training is evaluation of team skills* 
This paper describes methods and resources # and 
program considerations in team skills evaluation* 
The three areas pertaining to methods and 
resources arei development of evaluation 
criteria^ preparation of event scenarios^ and 
instructor training and additional resources* The 
program considerations include sequencing and 
coordination of team skills evaluation in the 
context of an overall operator training program* 



Team training has received considerable attention from 
practitioners and researchers. However, much less attention has 
been paid to developing, practicing, and evaluating team skills. 
Rather, teams are trained as a unit with the focus on technical 
skills but without explicit consideration of nontechnical skills 
such as communications and effective influence* 

An approach to team skills training for control room crews 
has been proposed based on an extensive literature review and a 
workshop attended by team training experts *^*2 £iye phases of 

the proposed approach are shown in Figure 1* The focus of this 
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paper is on the fourth phase of the approach. Team Skills 
Evaluation, Since diagnosis is a team skill that has received 
considerable attention in the Industry, diagnostic activities are 
also considered in this piper, 

mfALUATION METHODS AND RESOUmCSS 

The evaluation of knowledge regarding team skills can be 
assessed affectively using paper-and==pencil tests following 
classroom presentations* EKamples are provided in Figure 2. The 
evaluation of team skills is a somewhat more challenging 
undertaking. An evaluation of team skills will likely take place 
in a simulator or other simulation such as a control room 
mockup. The evaluation of team skills in the dynamic conteKt of a 
simulator requires development of criteria^ preparation of 
scenarios^ and instructor training and resources. These three 
considerations are discussed further in sections that follow. 

Development of Evaluation Criteria 

Evaluation criteria must be developed based on the learning 
objTOtives for the simulator training program. Objectives and 
criteria can be specified at two levels - generic and operational 
team skills. Generic team skills are the skills that apply to 
team performance regardless of the industrial setting. Generic 
team skills include communications, feedback, effective influence, 
conflict resulution, and leadership. To assist in the evaluation 
of generic team skills in the simulator, operational team skills 
can be used as exai^les. Operational team skills are situation- 
specific interaction requirements, A sample section of a 
checklist for crew evaluation is provlda^f in Figure 3, Sources of 
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information for developing operational team skill eKan^las include 
job analysis data, and positive and negative critical incidents 
from past plant operation. 

The generic team skills and acconqpanying operational team 
skill examples can be prepared in a checklist form for simulator 
instructors to use during observation and debriefing. 
Identification of specific points in time during a scenario at 
which specific team skills can be observed is difficult because of 
the dynamic nature of crew activities. However, a review of the 
scenario by the instructors, before the eKercise, can identify 
critical or complex points during the exercise at which effective 
team skills are most important. Cues or flags for the instructor 
can then be inserted at these points in the technical portion of 
the simulator lesson plan. 

Preparation of Event Scenarios 

Simulator scenarios for use in team skills practice and 

evaluation can be developed to target critical, complex, or 

freguent crew interactions in which team skills are particularly 
important for crew performance. 

Ideas for scenarios that can be developed are listed in 
Figure 4, These t^es of events often involve requirements for 
crew interactions that are complex, critical, and frequent. Thus, 
scenarios that have the characteristics t T,ted in Figure 4 usually 
provide good opportunities for observati . of team skills. 

Instructor Training and Additional Resources 

Simulator instructor skills of observation and debriefing are 
particularly critical for team skills evaluation since many of the 
exan^les of operational team skills must be inferred from 
observable crew performance, and checked for accuracy during 
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debriefing* Simulator instructors must observe both the technical 
and nontachniqal skills of the crew. 

During the observation, one important consideration is 
maintaininf realism so that crew interactions are performed as 
they would be during actual plant operation. If, for eKSmple, 
communications within, or to points outside, the control room are 
not simulated realistically^ the crew and the instructors will not 
have the oj^ortunity to observe this team skill* 

Instructor intervention during a simulator exercise is 
approprit."^ in certain situations. However, the instructor may 
want to avoid interjecting comnents for the crew, or to avoid 
freezing an eKercise before completion of a task, so that the 
instructor can observe realistic crew interactions. 

During the debriefing, a nondirective approach has been 

- 1 2 - 

reaommended. ^ The nondirective approach involves the instructor 
asking an open=ended question to begin the debriefing, and posing 
additional questions as needed for a comprehensive debrief* 
However, most of the debrief can be conducted by the crew members 
themselves* The crew should be given the opportunity to describe 
their performance and e^^lain their perceptions of what 
occurred. Crew members should be encouraged to use good feedback 
skills among themselves and reinforce good performance- If the 
crew experienced difficulties, they should be encouraged to 
generate strategies for overcoming the problem in future identical 
or similar operational sequences. The advantages of the non- 
directive approach are numerous i crew mergers are usually self- 
critical and often provide insights and criticisms that the 
simulator instructor might not have been able to observe or 
interpret, crew members will often accept the peer critique and 
self critique with less def ensiveness than if the suggestions come 
from an "outsider," and crew members may be more likely to accept 
strategies for overcoming problems in an eKercise if they generate 
the strategies. Thus, the instructor's role is one of 
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facilitator. The instructor may find the use of videotapes of 
selected portions of the eKercise helpful in stimulating crew 
discussions of the scenario. Simulator computer printouts from 
the eKercise can also be useful as a reference aid to the crew 
during debriefing. 

PROGRAM CONSIDERATIONS 

Program Gonsiderations in team skills training include 
sequencing and GOordination of team skills evaluation with other 
training. Team skills training and evaluation may be most 
effective if individual crew mambars are proficient in the 
technical skills and knowledge required for their jobs* Team 
skills can than be practiced and evaluated to improve coordination 
among the individual crew members. 

After an initial introduction to team skills for 
raqualif ication crews during a retraining session # subseguent 
requalif ication sessions may include a review of team skills and 
practice targeted to plant-specific problems encountered in using 
team skills on shift. 

Team skills evaluation can be coordinated with related non-* 
technical training evaluation in areas such as supervisory skills, 
communication skills # and diagnostic skills, to avoid unnecessary 
overlap or potentially confusing differences in jargon among 
similar courses. Team skills evaluation can also be coordinated 
with relevant technical skills training evaluation such as 
eriiergency operating procedures training or mitigation of core 
damage training* 
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SUtfldARY AMD CONCLUSIONS 



h systematic approach to team skills training for control 
room crews has been proposed recently, One phase in the 
approach is team skills eyaluation. This paper has discussed 
considarations in the simulator evaluation of control room crew 
team skills. The development of evaluation criteria, preparation 
of scenarios, and instructor training and additional resources 
have been discussed* The program issues of sequencing and 
coordination of team skills evaluation in the content of an 
overall operator training program were also described. 

The team skills training area is a relatively new focus for 
control room crew training. Ongoing research and development 
efforts in the industry will contribute to improved team skills 
training in the future. As examples, the Electric Power Research 
Institute is sponsoring research on a simulator instructor 
training module and on operator reliability models, the institute 
of Nuclear Power C^erations is continuing work in the area of 
developing teamwork and diagnostic skills, and the U,s. Nuclear 
Regulatory tommission is sponsoring research on operator cognitive 
modeling. These efforts will provide a greater knowledge base of 
control room crew performance and training from which to refine 
team skills training and evaluation. 
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TEAM SKILL OBJECTIVES 



Identl^ Generic 
Teim Skills 
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TiAM TASK TRAINING 
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Training 
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EVALUATION 
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PROGRAM 
EVALUATION 



Fig, 1. Approach to Team Skills Training 
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EKamples ©f Knowledg© Required 
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Communieatlon and feedback 



Effective influence 



ConfliGt management 

Leadgcship 
Lag nostic 

Situation a^raisml 
Problem diagnoiis 
Planning action 
Action and followup 



1 Illation Criteria I 



Communication model 
Communication bacr ien 

Inquif y 
Advoca^ 

Approach and avoidance 
Negotiation 

Roles of leader 
Leadership styles 



Pfoblem recognition 
Assessment of priority 

Identifiication potential causes 
Tactics to assist problem diagnosis 

Identification of action alternative 
Evaluation of action alternatives 

Analysis of effectiveness 
Recovery from misdiagnosis 



Test item for each objective 



Exchange information 
provide perforniance feedbd 

Ask question for resolutic 
Assert position 

Address problems 
Achieve solutions 

Delegate tasks 
Develop strategies 



Collect information 
Evaluate information 

Monitor critical safety fui 
Consider wide range possib: 

Identify procedure path 
Compare options 

Take appropriate action 
Consider contingency plan 



Checklist item with observa 
behaviors for each objectiv 



Fig. 2. Overview of Svaluation Approach for Team and Diagnostic Skills 
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2. Feedback 

Team membars raceived feadback from other 
team members which helped them to deter- 
mine the appropriateness and effectiveness 
of their actions. 



YES NO N/A 



Exaj^les I 

a. Accuracy of information was evalu-- 
ated by asking questions, inter- 
preting the information, and 
verifying the information with 
other team members* 



Performance feedback was given that 
enabled team mentoers to maintain 
appropriate actions or to correct 
deficient performance. 

Both positive and negative feedback 
were used. 

Feedback ymm given as soon after the 
behavior as possible. 



Comments : 



Fig, 3* Sap^le Portion of Team Skills Evaluation Checklist 

Page 1 of 2 
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YEE NO N/A 

3. Effect iva Influence 

Team memberE successfully got their views 
across and persuadad others that certain 
action should be taken. 



Examples i 



Team members designated other team 
members to perform specific tasks. 

Team members asked questions to 
get information reguired to estab- 
lish a position or resolve a 
question . 

Team members assertively stated 
and defended their positions. 

When involved in group problem 
solving and decision making f team 
members asked questions, obtained 
additional information, and stated 
their opinions. 



Coimnen ts ; 



Fig* 3* SaiT^le Portion of Team Skills Evaluation Checklist 

Page 2 of 2 
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o Single Possibla Cause 

= One Non^Obvious Action 

Multiple Obvious AGtions 
- Multiple Non-=Obvious AGtions 

o Multiple Possible Causes 

~ toe Non^Obvious AGtion 

" Multiple Obvious Actions 

~ Multiple Non-Obvioua Actions 

o Syn^toms of Event Closely Resemble Those of a Different Event 
Q Low Probability Event with Serious Consequences 
o Tlme-Ciritical Events 

o Necessity for Consideration of plant HistoriGal Data (Past 
Few Hours of Operation) 

o Instrumentation Failure 

□ Confederate on Crew 

Aerator Withholds Information 
- Sabotage 



Fig, 4. Typmm of Operating Event Scenarios for 
Team skills Training and Evaluation 



_DEVELOPMENT OF BWR AND PWR EVENT DESCRIPTI ONS 
FOR NUCLEAR FACILITY SIMULATOR TRAINING * 

R. J, Carter 
C. R, Bovall 

ABSTRACT 

A number of tools that can aid nuclear facility 
training developers In designing realistic simulator 
scenarios have been developed. This paper describes each of 
the tools, i.e., event lists, events-by--competencles 
matrices, and event descriptions, and Illustrates how the 
tools can be used to construct scenarios, 

INTRODUCTION 
Background 

The U.S. Nuclear Regulatory Commission (NRC) is responsible for: 
prescribing uniform conditions for licensing individuals as operators 
of nuclear production and utilization facilities; determining the 
qualifications of these individuals , and Issuing licenses to such 
individuals (Ref. i). This operator licensing system is comprised of 
both a written and an operating examination. The operating exam is 
further divided Into two parts, oral and simulator. These three 
examinations are oriented towards determining whether the applicant 
for an operator's license has learned to operate a facility 
competentJy and safely, and additionally. In the case of a senior 
reactor operator (SRO), whether the applicant has learned to direct 
the activities of licensed oparators cnmpetently and safely. 

* The research was sponsored by the NRC under U.S. Department of 
Energy (DOE) interagency agreement 40-550-75 with Martin Marietta 
Energy Systems, Inc. under contract no. DE-AC05--84R21400 with DOE. 
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Guidance to the facility licensee In regards to the slcsulator 
exam is detailed In paragraph 23 of part 65 to title 10 of the Code of 
Federal Regulations (Ref. 2). it states that the simulator 
ejcamlnatlons for reactor operator (RO) and SRD applicants are 
generally similar in scope and require each applicant to demonstrate 
an understanding of and the ability to perform the actions necessary 
to accomplish a list of 13 Items. Paragraph 23 also says that the 
content Is Identified, in part, from information In the final safety 
analysis report, operating manuals, facility license and license 
amendments, licensee event reports, and learning objectives derived 
from a systematic training analysis performed by each facility 
licensee . 

NUREG-IOZJ (Ref. 3) provides the policy and guidance to NRC 
operator licensing examiners and establishes the procedures and 
practicea for the examination and licensing of applicants for NRC 
operator licenses. It is Intended to assist NRC examiners and 
facility licensees to understand the exam process better and to 
provide equitable and consistent administration of sxamlnatlons to all 
candidates for either RO or SRO licenses by NRC examiners. Guidance 
and policy on the administration, scope, and objectives of the 
operating and simulator exams are detailed in examiner standards (ES) 
301 - 305 and 501 - 502, respectively. 

Overview 

In 1982. the Office of Nuclear Reactor Regulation at the NRC 
started a program which is oriented towards improving the validity of 
the operator licensing examination and the reliability of the exam 
process. Spilberg described the project and the Issues and problems 
which are being addressed at a previous Training of Nuclear Facility 
Personnel symposium (Ref. 4). Oak Ridge National Laboratory has 
recently completed a research project which was peformed in support of 
this NRC program. The purpose of the effort was to develop a set of 
tools for examiners to use during the construction of scenarios for 
boiling-water reactor (BWR) and pressurlzed-water reactor (BWR) 
slmmator exams. The focus of the project was on the generation of 
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BWR and PWR event Hsts, a mapping of the competencies which are 
scored on the simulator examination to the events* and the design of 
of f -normal , I.e., abnormal and emergency, event descriptions. While 
these tools were created for use by the NRC in operator licensing, 
they seem to be applicable to the nuclear power Industry as a whole 
and can be used by a facility's training department in Its design of 
scenarios for simulator training, 

SCENARIO PREPARATION TOOLS 

Event Lists 

Four event lists consisting of 87 events were constructed. The 
breakdown of these events in terms of reactor type and severity of 
event are as follows: 

a. BWR=abnormal = 26 

b. BWR-emergency - 22 

c. PWR-abnormal - 26 

d. PWR-emergency - 13 

The events were selected based on the following criteria: 

a. The event should be a significant casualty or abnormality. 

b. The event should be able to be replicated on the majority of 
plant-referenced simulators In use today, 

c. The event should be able to be effectively administered and 
evaluated within the time limits of a typical simulator exam. 

d. The e\^ent should provide a useful base upon which to evaluate 
candidate eligibility for licensure. 

The source data for the generation of the event lists consisted of the 
Institute for Nuclear Power Operations (INPO) job^task analysis, 
American National Standard 3.5 (Ref. 5), an event list derived during 
an NRC eKaminers workshop*** emergency procedure guidelines (EPG), and 
other related references. Tables 1 4 present the four event 
lists - 

Events^By^Competencles Matrices 

The eight competencies , as described in revision 2 to ES-302 in 
NUREG=^1021, were mapped to the BWR and PWR events. Each event was 
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Table 1, Bolllng^Water Reactor Abnormal Events 



Master Feedwater Controller Failure 
Nuclear Instrument Channel Failure 
Rod Position Indicating System Failure 
One Reactor Recirculation Pump Trip 
Trip of Both Recirculation Pumps 
Recirculation Pump Seal Failure 
Scoop Tube Lock 
Increasing Suppression Pool Temperature 
Drywell Cooler Failure 

Stuck Control Rod 
Uncoupled Control Rod 
Control Rod Drift 
Control Rod Drive Hydraulic Pump Trip 
Loss of All CRD Hydraulic Pumps 
CRD Flow Control Valve Failure 
Condensate or Condensate Booster Pump Trip 

Reactor Feedwater Pump Trip 
Loss of Feedwater Heater Extraction Steam 
Stator Cooling Water Pump Trip 
Steam Jet Air Ejector Malfunction 
Loss of One Reactor Protection System Bus 
Area Radiation Monitoring System Alarm 
High Main Steam Line Radiation 
High Ventilation Exhaust Radiation 
Inadvertant HPCI or RCIC Initiation 
Loss of One RBCCW Pump 
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Table 2. Boiling-Water Reactor Abnormal Events 



Reactor Scram With MSIVs Open 
Reactor Scram With MSIVs Closed 
Loss of Shutdown Cooling 
Gross Fuel Failure 
Excessive Reactor Cooldown Rate 
Anticipated Transient Without Scram 
Stuck Open Main Steam Safety/Relief Valve 
Small Break Loss of Coolant Accident 
Reactor Coolant Leakage Outside Primary Containment 
Jet Pump Failure 
High Suppression Pool Water Temperature 
Main Turbine or Generator Trip 
Main Turbine or Generator Trip Without Bypass Valves 
Loss of Condenser Circulating Water 

Loss of Feedwater System 
Loss of All High Pressure Feedwater 
Loss of Plant Control/Instrument Air 
EHC Pressure Regulator Failure (All Valves Open) 
Loss of Nuclear Service Water 
Loss of Reactor Building Closed Cooling Water System 
Loss of Off^Site Power 
Loss of All AC Power (Station Blackout) 
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Table 3. Pressurized^Water Reactor Abnormal Events 



Loss of RCS Makeup 
Loss of Automatic Pressurizer Pressure Control 
Failure of Pressurlzer Spray Valve 
Loss of Automatic Pressurlzer Level Control 
Progressive Failure of No . 1 Seal in RCP 
Failure of Steam Dump to Open 
Steam Generator Safety Valve Falls Open and Falls to Reseat 
Steam Generator Level Control Failure High/Low 
Dropped Control Rod 
Inoperable or Stuck Control Rod 
Inadvertant Boration at Pgwer 
Inadvertant Dilution at Power 
Failure of N^44 High 
Loss of Instrument Air 
Failure of Turbine to Runback Automatically and Manually 
Failure of Impulse Pressure Transmitter (Low) 
Steam Generator Tube Leak Within Capacity of Charging Pump 
Loss of Condenser Circulating Pump 
Crlticallty Outside Expected Band 
Failure of Loop Temperature Instrumentation High/Low 
Loss of One Main Feedwater Pump at High Power 
Spontaneous Opening of the Main Generator Output Breakers 
Loss of RCP Without Reactor Trip 
Main Steam Leak Inside Containment 
Rupture in Letdown Nonregenerat.i ve Heatexchanger to CCW 
Failure of Pressurlzer Control Bank Heaters 
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Table 4. Pressurizad-Water Reactor Emergency Events 



Reactor Trip 

Large Break LOCA - Reactor Trip With Safety Injection 
PZR/PORV Failure to Open 
Steam Generator Tube Rupture 
Failure of Main Turbine to Trip 
Small Break Loss of Coolant Accident 
Anticipated Transient Without Scram 
Loss of Auxiliary Feedwater - Inadequate Core Cooling 
Loss of Off-Site Power 
Station Blackout - Loss of All AC Power 
Control Room Fire Requiring Evacuation 
Main Steam Break Inside Containment 
RHR LOCA - Complete Loss of RHR 
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analyzed In an Iterative fashion by job position, i.e.. RO, SRO. and 
balance-=of -plant operator (BOP). The purposa of this analysis was to 
determine whether the event provided enough opportunity for the 
examiner to observe each of the competencies. Four events=-by= 
competencies matrices were arranged based upon the results of the 
analyses. In each matrix tJ\e events are the rows and the competencies 
are the columns; an "X** appears In a cell of the matrix If It was 
determined that a competency is exercised by a specific operator 
during an event. These matrices aid in the selection of a sufficient 
number of events and ensure that each candidate demonstrates each of 
the applicable competencies over the course of the simulator 
eKamlnatlon. A page from one of the competency matrices is shown in 
Table 5. 

Event Descriptions 

An event description of about 2-4 pages was prepared for each of 
the events. Each was written to be as generic. I.e., apply to many 
plants, as possible. The descriptions for the abnormal events were 
designed using available event-based plant procedures. The emergency 
event descriptions were developed using symptom-based EPGs from 
various owners groups (Refs. 6-9). Each event description is 
organized Into t^^ major parts: a cover sheet and a progression of 
operator action An example event description is exhibited as Table 
6. 

C over Sheet 

The cover sheet presents the following general information: 
1. Operating Sequence— The title of the event, 

^- Nuclear Steam Supply System Vendor/Reactpr Type— The nuclear 
steam supply system vendor{s), i.e.. General Electric* 
Westinghouse. Combustion Engineering, and/or Babcock & 
Wilcox, and the type of reactor. 



** The examiners workshop was held at the NRC headquarters in 
Bethesda. Maryland on August 8. 1985. 
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Table 5, A Paga from a Competency Matrix 
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Table 6* An Example Event Description 



Operating Sequence: Failure of Loop Temperature Instrumentation High/Low 
NSSS/Type r Westlnghouse/PWR 

Initial PLant State: Reactor Controls in Automatic Power Level at About 
75*, All Other Control Systems In Automatic 

Sequence Initiator: Loop (X) Hot Leg RTD (Narrow Range) .^ails High/Low 

Important Plant Parameters: 1) RCS Temperature/Pressure, Reactor 
Power, 3) PZR Level, 4) Rod Position 

Progression of Operator Actions: See Flow Chart 

Final Plant State: The reactor/turbine plant Is at steady state. The 
temperature defeat switches (delta T and Tave) in loop (x) are defeated. 
The affected loop blstables for overteinperature/overpower delta T have 
been placed in the tripped condition. 

Major Plant Systems: Rod Control, Reactor Proectlon and Control. RCS 

Tolerance Range i The reactor/turbine plant Is stable. The operator must 
place the rods in manual to mitigate the casualty. The blstables should 
be placed in the tripped position i the loop Tave and delta T Inputs 
should be defeated- 

Competencies Tested: 

SRC - Compliance/Use of Technical Specifications 
Supervisory Ability 

RO ^ Understanding/Interpretation of Annunciator/Alarm Signals 
Diagnosis of Events/Conditions Based on Signals/Readings 
Understanding of Instruraent/System Response 
Control Board Operation 

BOP - Control Board Operation 

NOTE: Most C-E units have similar system response, but operator response 
and corrective actions are different. 
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Table 6. Cont, 



FAILURE OF LOO? TEHPERATURE INSTRUMENTATTON HIGH/ LOW 
Pregresslen of Operator AcEionsj 



RO 




Aeknowl edge 
alarms 



<1) Failure High 
(2) No Red tfotion 

Due to Failure Low 



place rsd 
eentrel to 
ffl^nual 



SRO 



Refer Ce 
tech, speei 
R3£ pf PC , 
Qhannels 



RO 



TAVG S 
to defeat 
Defeated sn 
Cont. 

Brd. 



BOP/RQ 



Reetsre 




Stabilize 


TAVE-TREF 




reactor/ 

turbine 

pQwer 




■-I 



Inform Shift 
Supervlssr 



*AlarB5 

(1) Failure High (T^) 

- RCS TREF-TAUCT Hi/I^ 

- ECS DevlatlQn 

- TAVG-TREF Deviation 

- OT Trip Alert 

- QPJkT Trip Alert 

(2) Failure Low (T^) 

' RCS AT Deviatidn 

- RCS TREF-TAUCT Hl/Lo 

^ Pesslblt Rod Insertion 
Limit Alarm 



SHO 



RO 



Notify I&C 




Trip OP. a 
OT. ^T 
blstables 




Return rod 
control to 
automatic 














SRO 


SEO \ ISC 





Notify 




Plant 




Hmnsgmt . 




per 




procedure 



Initiate 
jfepalrg 
Report/ 
Eecsrd 



349 



V.A.3,12 



I n 1 1 j a I_P I an t^t ate - - The operating status of the 'plant at the 
time the event starts. The Initial plant state may be 
obtained either by use of the Initial conditions input into 
the simulator computer, or by instructing the candidates to 
take the plant Into the desired state. 

Sequence Initiator --A brief description of the equipment 
fallure(s) that causes the event. 

Important Plant Parnm^tf-r^^^Thn^^ plant parameters that 
should be monitored by candidates during the course of the 
event. Only parameters which are unique to the event are 
listed; parameters that are Important In virtually every 
off-normal condition, such as primary system pressure, water 
levels, and reactor power, are not repeated in each 
description. The important plant parameters are Intended to 
provide objective bases for use in candidate evaluations. 
Including the ability to diagnose plant conditions, comply 
with procedures, and observe technical specification limits. 
Progression of Operator Events — This Is discussed below. 
Major PI ant^SQ^ems— Those pJant systems that are uniquely 
affected by the event. The plant systems which are listed 
either experience the fallure(s) or are used In mitigating 
the consequences of the failure(s). 

Tolerance Range— The tolerance range of operator actions 
represents the bounds within which the candidates must 
respond before the technical limits are exceeded. Similar to 
•'important plant parameters", tolerance ranges are intended 
to nrr ide objective bases for use In candidate evaluation. 
Final Plant State—The possible plant conditions by which a 
Judgment can be made to end the event and move on to the next 
part of the scenario/examination. The event may he ended 

before this point Is reached provided that enough information 

has been gathered to adequately assess candidate performance. 

However, if time permits, the event should be taken to the 

indicated final plant state. 

Competen cies Tested —This was discussed above. 
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Prog ress l^ n of Operator Actions 

The progression of operator actions depicts in a flow chart 
mannar the representative sequence(s) of expected Imniedlate and 
subsequent candidate actions, including communication, that can be 
observed during the event. These flow charts are Intended to be as 
generic as possible for a given reactor/vendor type. The flow chart.^ 
indicate that, in some cases, there is more than one path which the 
event can take. The path taken will depend on the likely pertubations 
of the system, the decisions of the candidates, and/or choices made by 
the examiner. The objective of these multiple paths is to provide as 
much flexibility as possible, while retaining simplicity. 

SCENARIO PREPARATTnN 

Development of effective scenarios using the event descriptions 
is a five-step process: 

1. Selection of the events. 

2. Listing of the events. 

3. Completion of the simulator scenario form. 

4. Completion of the simulator administration form. 

5. Completion of the competency checklist. 

Step One: SelRnt jon of the Even ts 

The event descriptions are Intended to aid the examiner In 
selecting simulator events for compliance with the criteria described 
in ES^302. These criteria include: 

1. Events requiring candidates to operate in normal evolutions. 
Instrument failures, component failures, and major plant 
transients . 

2. Events requiring candidates to operate under a range of 
conditions within each category as listed in Item #1 above, 
such as degraded heat removal, degraded electrical power, 
containment challenges, and degraded pressure control. 

3* Events that Impact Important safety systems such as the 
systems Identified in PWR/BWR knowledge and ability 
catalogues, i.e., NUREGS-1122/1123 (Refs. 10 & 11). 
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4. Events that, together, will provide ample opportunity to 
evaJuate each candjdate on each relevant candidate 
competency , 

5. Events that will complement and/or supplement Information 
gained on the candidates during the written and oral 
examinations , 

The events-by-competencJes matrices shouJd be used as an aid during 
the selection of events. They will be helpful In choosing a 
sufficient number of events and ensuring that the candidates 
demonstrate each of the applicable competencies over the course of the 
simuiator exam. At a minimum, enough events shouJd be selected so 
that each competency Is demonstrated at least once more than once. 
Step Two: Listing of the Events 

Each exam scenario should present the candidate with a logical 
and reaJistic set of probjems to which he/she is to respond. For 
example, component and instrument failures can be used as precursors 
to major casuaities. This win fulflJl two or three exemlnation 
requirements while achieving scenario realism. A rough list of the 
events that are to be used in each scenario shouid be made. The 
events should be placed in a sequence which is logical and in which 
they will be initiated during the scenario. 

Step T hree: Completion of the Simulator Scenario Form 

The simulator scenario form (ES=302, Attachment 3) provides the 
simulator operator with a set of Instructions for entering Initial 
conditions and malfunctions into the simuJator computer. The 
Information for this form is obtained from the event description cover 
sheets and simuJator reference materials, partlcuJarJy the initial 
conditions and the malfunction cause-and^ef f ect descriptions. The 
cover sheets are useful for providing information on the inltiai plant 
state for a given event and the simulator malfunctions that may be 
used to initiate the event. The first item is to seiect the 
appropriate plant condition from the initial conditions menu. For 
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example, If the event description specifies that the event should be 
Initiated from high power, an initial condition for this power level 
may be selected from the menu, or a Jower power level may be SGlected 
and the candidates directed to perform a power escalation. This will 
meet the requirements for a normal evolution or reactivity change, and 
a major casualty. The malfunctions to be run during the scenario, 
along with the elapsed time that the maifunctlon shouid be initiated, 
should then be Included on the simulator scenario form. 

Step Four: Completi on of the Simula tor Admini stration Form 

The simulator administration form (ES-302, Attachment 5) should 
include the Qbservable candidate behaviors for use in evaluating 
candidates. The progression of operator actions can be used as an aid 
in developing these expected actions/behaviors. This information 
should be compared to the plant specific procedures and technical 
specifications to ensure the appropriateness of the flow chart 
Information for that facility. 

Each action block in the flow charts indicates the candidate 
primarily responsible for the action. This is intended as a guide and 
may not be accurate for every situation. In general* the SRO is 
responsible for directing the actions of the RO and the BOP, 
communication with the auxiliary operator and other support personnel, 
and all administrative duties. The RO is primarily responsible for 
the reactor and reactor auxiliaries within easy reach of the reactor 
panel. The BOP is responsible for all plant secondary systems, 
electrical distribution, emergency core cooling systems, and 
process/area radiation monitoring. However, when the workload on one 
operator becomes excessive, assistance may be given by another 
operator. When an action Is entered on the simulator administration 
form, the candidate responsible for the action is indicated in the 
"position" column. 

Step Five I Completion of the Comp ete ncy Checklist 

After the first scenario has been drafted, the expected actions/ 
behaviors listed on the simulator administration form should be 
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reviewed, along wlrh the competencies tested which are Identified on 
the event dnHcrJptlon cover sheet or the applJcab]o events-by- 
competenciea matrix, to determine which competencies should be 
addresBed for each candidate. Subsequently, these competencies should 
be entered onto the competency checklist (ES=302, Attachment 8). If 
the checklist contains competencies that have not been checked off, 
the selection of events for the next scenarios should be chosen, In 
part, to evaluate these competencies. 
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MAINTENANCE TRAINING ■ A MODERN NECESSITY 



William Buahall 



ABSTRACT 

In recent years ^ there has been an increase in technically 
advanced systems and equipment and a need for highly skilled 
and knowledgeable maintenance technicians to maintain them. 
To Implement an effective training program^ training groups* 
and plant staffs' key Co success must be cooperation and 
creativity. This paper deals with plant staff interface and 
how to effectively conduct perf ormance^based training while 
holding the line on costs* 

This paper includes cost effective and innovative measures 
to produce performance-based training for maintenance dis-- 
cip lines including i 

" Using the plant staff as a resource as subject 
matter experta in the development and verifica^ 
tlon of training materials* 

^ Using the plant staff as a resource for the 
construction of training aids. 

Using salvage and surplus to produce high quality^ 
low cost training aids. 

Using cutaways for better understanding of the 
theory of equipment operation. 

These cost saving practices are currently being used at 
Gulf States Utilities- River Bend Nuclear Station* 



V.A.4.1 



355 



MAINTENANCE TRAINING - A MODERN NECESSITY 



The Plant Scrams the lights go out. Minutes seem like hours 
as Technicians scramble to restore the system. Operators are idle, as 
maintenance proceeds. The phone calls and questions begin What*s 
the problem?'S ^'When will we be back up?", "This is costing us a fortune!" 

The next day the inquisition is held. Why did it take so long to 
make the repairs and get the system back up? The Technicians were train- 
ed, weren^t they I? Eyes turn to the Trainers. It seems that training on 
this particular equipment was planned last month but was cancelled due to 
"overhead", budget cuts, lack of manpower. 

We know through data collected over the past few years that maint- 
enance training is a necessity and that it has increased effectiveness in 
the field, reduced down time and saved money. This training is necessary 
due to modern advances in technology and complexities of the equipment 
and systems throughout the industry. 

The man-hours and dollars expended on downtime and lost customer 
service certainly aren't considered overhead! How can you provide Maint- 
enance Technicians the effective knowledge and performance training needed 
to troubleshoot and repair sophisticated equipment and systems AND stay 
within your training budget? The answeri 

USING THE PLANT STAFF, AS A RESOURCE FOR SUBJECT MATTER EXPERTS, 
IN THE DEVELOPMENT AND VERIFICATION OF TRAINING MATERIALS. 

USING THE PLANT STAFF AS A RESOURCE FOR THE CONSTRUCTION 
TRAINING AIDS. 

USING SALVAGE AND SURPLUS TO PRODUCE HIGH QUALITY, lA^l-^ t^\./.: 
TRAINING AIDS, 

USING CUTAWAYS FOR BETTER UNDERSTANDING OF THE THEORY OF EQUIPMENT 
OPERATION, 

Who better to obtain Information from, for training materials devel- 
opmant, than the plant system experts | both engineering or technician types 
who have first hand eKperience of the system or component. This is a 
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valuable resource g as no one person can be an expert on all components 
or systems 5 due to modern compleKitles * These experts do not have the 
time to be training InsCructors* By utilizing thems as a source of In" 
formation and verification, many hidden problems may be discovered and 
a better understanding of the systems may be included in the training 
material for presentation in the classroom* 

Another factor to be considered is that many companies rely heavily 
on vendors I in many cases this is very goodp but you must realize that 
their knowledge and expertise is on a component and not on the plant 
specific system application in which the component is installed. We 
have all found manufacturer's literature to be vague and generic, since 
it is written for multiple applications. The plant staff subject matter 
experts ''SME", is familiar with the plant specific application and can 
provide more pertinent information for your training material* 

Direct interface between training and plant staff should be on a 
"one on one" basis between the SME and the instructor p which is generally 
volunteered without hesitation. For verification of both vendor and in^ 
house training, and to add operating experiences and ensure correct Infor 
matlon dissemination, class attendees should be comprised of experienced 
personnel, as well as non^^exper lenced personnel* Consider also these 
SME's as guest lecturers and material developers for lesson plans on the 
more complex equipment and systems, which are in use* Do not neglect 
considering operations personnel for either development or lectures 5 this 
gives another point of view of systems and operating experiences , espe^ 
clally in regard to troubleshooting and analysis* 

Sources for seminars, outside of plant staff, may be provided "free 
for the asking" from local dealers or distributors* These seminars are 
excellent for that extra information on speciality items ^ eg| heatshrinks 
fittings, fuses, bearings 5 seals 5 and consumable items* 

After training material has been developed to present theory and 
knowledge concepts to the trainee 1 the next step would be to produce or 
procure equipment and material for the performance portion of the train* 
ing. 
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Remembering we have a very limited budget, the closest source of 
supplies would be the plant itself. 

Plant staff personnel can produce needed training aide, such as 
mock- ^3 and stands from salvaged, damaged out of spec, QA rejects or 
sera,., ,d equlpmBnt and materials i also to be considered is material 
considered excess, or from decommissioned or canceled units. 

Even though plant staff manpower is limited at times, training is 
a continuous program. There are slack work periods in the plant, during 
which cutaway machining, welding and fabrication of training aids can be 
accomplished. Utilitzation of the trainees to build or erect the trainin 
aids also serves a dual purpose i this exercise provides knowledge of the 
actual construction of the training aid and gives the trainee practical 
performance training. 

The use of salvaged or surplus material as a training aid requires 
nothing more than an understanding of the company's procurement and 
accounting procedures. Gaining this understanding may be a complicated 
and time consuming process, but the reward is procurement of Inexpensive 
training aids that are plant specific. Understanding procurement and 
accounting procedures enables those involved to become more familiar 
with these procedures and policies and helps them to recognize what items 
may be available within their own companies as well as from other sources. 

Utilization of existing materials and facilities only takes a few 
moments of time. Now have you thought of the easiest places to obtain 
materials? These are the dump, bone yards, junk pile or garbage heap, all 
you have to do is go and look you never knows and how about the manufact- 
urer, local dlstrlbucors or agents, who assisted during constructioni 
those that have contracts with the plant will usually donate scraps or 
sample items for training. These are also very good sources of infor- 
mation as well for providing short seminars or demonstrations. 

Cutaways are an invaluable training aid. They provide a better 
understanding of equipment operation, especially for complicated and 
multi-component constructed equipment | by showing the physical internal 
construction of the component, eg, fuel injectors, multi-part valves. 
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gear boxes and pumps. Cutai^^ays allow the technician to actually see how 
the theoretical concept of the item is utilized In that application. 

All of these training aids help the technician to actually see, 
feel and perform specific tasks associated with their job in an effort 
to increase their effectiveness in the field in making quick and 
accurate repairs* 

By utilizing the company's assets which are sometimes overlooked or 
forgotten^ this training may be completed under budget* 

Attached is a list of items s procured through the methods described 
In this paper 5 with apprOKimate savings incurred. 

I would like to acknowledge the assistance given by Leroy East, 
Chuck Campbell and Jimmy Allen for their effort in the identification 
and procurement of many of the listed training aids. 
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SQUmilNT PROCURED ESTIMATED SAVINGS 

^ — — — TO TRAINING 

480V Swlcehgear with breaker and relays 

4160V Switchgear with relays class lE lost 
QA due to damage 

Class IE battery cells in hot storage ^ - ^ 

Con loan) " " - '-0 

Welder and rod oven 

200 hp motor St pump assembly 

25 hp motor 

Cummings Diesel Fire Pump complete 

Control Rod Drive Mechanism and nnn nn 

Hydraulic Power Skid 50,000.00 

Vertical multi-scage pump 
Small Air Compressor 



$ 3 5 000. 00 
6*000,00 



2p000,00 
2,000.00 
500.00 
ISpOOO.OO 



500,00 
500.00 

Recirc Pump Motor Stand and Seal nnn nn 

ituffing Box w/tools 60,000,00 

Various small valves 

EMD Diesel Fuel Injectors 

Pipe threader. Drill press 

Out of Cal M&TE, Recorders, Misc Transmitters, 
used trip unit cards 

1200 HP Vertical Motor 

EQUIPMENT IN PROCESS OF BEING PROCURED 
SMBOOO MOV's 
Nash Vacuum Pump 
Safety Relief Valve 
Testable Check Valve 

ITEMS BU ILT BY PLANT STAFF PERSONNEL 
Alignment mockups 
Cutaway Fuel Injectors 
Valve Cutaways 
Control Rod Drive Tools 



500.00 
2,000.00 
1,000.00 
5,000,00 

60,000.00 
$218,000.00 



5,000,00 
25,000.00 
10,000.00 

5,000,00 
45,000.00 



2,000.00 
1,000,00 
1,000.00 
3,000.00 
i 7,000.00 

TOTAL ESTIMATED SAVINGS 1270,000.00 



3G0 



The following paper was not received In time to be published in the pro- 
ceedings , Space is provided below for notes . 



Malntenanca Trainingi An Historical Perspective,, 
George Shaw, Rochester Gas and Electric 



NOTES • 
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EFFECTIME TRAINING FOR THE NUCLEAR FIELD USING COMPUTERS 



S ^ E ^ Farrmr 

G, Dickelman 



ABSTRACT 

Since the introduction of microcomputers there 
have been training applications in many 
disciplinesi the nuclear field is no eHception* 
Because of the use of computers in training, a 
variety of development tools known as author i no 
systems have emerged for creating instruction. 
There is clearly a connect ion beti^een the 
capabilities of the authoring systems and the 
effectiveness of the training they produce. 
Unfortunately? it is too often the case that the 
limitations of the authoring systems drive the 
instructional design of the programs they produce. 
The result is very poor or limited instruction. 

This paper deals with the instructional design 
of good nuclear training material and its 
implications for features that must eHist in an 
authoring system. The paper looks at four 
categories^ visual displays , sound f i nterac t i on s 
and rea 1 i sm . Each category is examined from the 
standpoint of training program content and 
instructional designf then the authoring system 
features that are required to support each category 
are determined. Consideration is given not only to 
the list of features^ but also to their 
integration? ease of use? and flexibility. 

The reader will ultimately be provided with a 
good sense of which features and characteristics 
must enist in an authoring system in order to 
produce ef f ec t i ve nuc 1 ear tra i ni ng mater i a 1 , In 
particulars the reader will find useful information 
to apply to the process of authoring system 
se 1 ec t i on » 
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Computers have been used in education and training 
for many years . 1 1 wou Id seem to follow that the task of 
adopting computers for your training material would be an 
easy one. This might be the case if computer technology 
grew at a relatively slow rate? but this is not the case. 
Since 19S1 there have been a tremendous number of changes 
What follows are instructional tools and techniques which 
take advantage of the technological changes. The result 
is an environment which is extremely rich in alternatives 
for creating? delivering^ and managing instruction on the 
computer ^ 

Instructional technology consists of instructional 
strategies and proceduress softwaref and hardware. The 
primary focus of this paper is on the software and how it 
can support your des i gns-- those which support training in 
the nuclear field. The name given to software products 
which support the lesson creation process is author i ng 
system. The followirfg sections define and discuss thi^ 
term? then determine features which must exist in an 

authoring system in order to support effective nuclear 
training material* While the ideas discussed here are 

applicable to most computer systems, the focus will be 

primarily on the microcomputer since it currently has the 

widest application in training, 

AUTHORING SYSTEMS - WHAT ARE THEV? 

Loosely speaking? an authoring system is a collection 
of software tools which assists in the creation? delivery, 
and management of compu ter— based instruction* While 
general purpose computer languages (BABICi FDRTRANj 
PASCAL, C) can be used to create instruction it is 
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recognized that they lack the specialization for 
instructional designi it is tedious, time consuming* and 
therefore costly to create instruction using general 
purpose languages. 

In particular J an authoring system provides the meani 
to create display frames or screens? link the frames in 
some order f provide the opportunity for student 
interaction? process and respoind to student input? and 
manage the process. The degree to which programming is 
necessary varies from system to system, but it is 
generally the case that the features they provide 
substantially reduce the time and effort associated with 
creating instruction* Table 1 gives a general list of 
features which define an authoring system. 

Table 1. General Features of an Authoring System 



Frame Creation (TeKt t Graphics Editors) 
Frame Logic (Branching) 
Ouestion and Anst^mr Creation 
I nt er ac t i on 
Lesson Debugging 
Student Data Storage 
Program Management 



Language System Versus Menu^Dr i ven System 

There are two opposing designs for authoring systems^ 
language systems versus menu--driven systems. Language 
systems are programming languages which have the specialty 
items of Table 1 built in| menu=driven systems? as the 
name impliesi require no programming. Each approach has 
its merits and pitfalls. 
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Generally 5peak*ng5 menu--dr i ven systsms are easy to 
use but are restrictive because, j^ithout prog r amm i ng 5 you 
canno t make use of many at the computer's capabilities. 
Language systems are rich with cap^.b i 1 i t i es but are more 
complicated and time consuming to use because programming 
is involved, A greater level af computer expertise is 
required to use language systems than menu^driven systems, 
but the end product can generally be mare sophisticated if 
language features are utilised to created certain effects. 

An emerging alternative to these approaches is the 
hybrid authoring system i^hich incorporates the best of 
both worlds. That is, it has a variety of levels at which 
it may be used\ The "novice" level is primarily a menu-- 
driven system, but "expert" levels allow the user to 
access a language (or languages) for applications in which 
the menus do not provide sufficient alternatives. The 
best hybrid system will adapt to your level of 
sophistication to the extent that you can "drop out" to a 
general purpose language for the most demanding effects 
while still using menu-driven editors to perform routine 
tasks. Figure 1 illustrates the distinction between menu- 
driven and language authoring systems and their 
relationship to the hybrid system. 



MENU^DRIVEN SYSTEMS LANGUAGE SYSTEMS 



HYBRID 
SYSTEMS 



Figure 1; Menu-Drivens Language, and Hybrid Authoring 
Systems 
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AUTHQRINQ SYSTEMS FOR THE NUCLEAR FIELD 

In order to select the proper authoring tool -for any 
training application you must "first examine the content 
dofnain and instructional design of the program. It is too 
often the case that training organizations attempt to 
select the *'best" authoring system for compu ter=based 
pragrams by comparing authoring systems r.ather than by 
considering the programs they wish to deliver by computer. 
The result of this approach? unf or tuna tel y , is often the 
selection of an authoring system which restricts, or aven 
dictates^ the instructional design of computer —based 
programs. (By definition? a system is not the universe 
and will therefore contain Timitationsi choosing the wrong 
set of limitations can be disastrous to instructional 
design * ) 

Training programs for the nuclear field are numerous 
and vary widely in content and design. The ultimate 
mission of the training? however,? is the safe operation of 
nuclear generating stations- Whether a program trains 
operators on the use of safety injection systems? how to 
safely approach criticality? or emergency procedures in 
the cfivse of a steam generator tube rupture? there are 
specific standards that a computer-abased program must meet 
in order to satisfy the mission* In particular the 
programs must create meaningful v i sua 1 d i sp 1 a_y s ? in some 
cases reproduce actual sound ? provide meaningful 
i n t er ac t_i on ? and must be r ea 1 i s t i c in content. The 
following sectiofis define precisely what these mean in 
terms of authoring system capab i 1 i t i es * 
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Visual Displays 

Nuclear training material is demanding with regard to 
visual displayB. When an aperator first learna a 
mechanical or .lectricai symtem, he or she must become 
familiar with conceptual line drawings (PilDs) as well as 
identify physical panels and displays. A>-thoring software 
must therefore provide the capability to produce graphics 
with good resolution at a reasonable cost in terms of time 
and effort. Ideally, the authoring system should be 
flexible enough to import graphics created .rith other 
popular graphics programs and provide the means to modify 
the imported graphicB. 

The authoring system must allow for the creation and 
display of custom text characters. Trainers in the 
nuclear field ere routinely frustrated by the absence of 
the industry-standard characters used in nuclear science 
(reactivity, delayed neutron fraction, chemical symbols, 
etc.) on microcomputer displays and word processing 
packages. The graphics mode of all standard 
micracomputers is fully capable of displaying custom 
characters and fonts and allowing for their creation via 
special software programs. At a minimum, the authoring 
system you select for nuclear training applications should 
support custom characters and fonts. 

The authoring system must also allow for the 
combination of tent and graphics. That is, text and 
graphics are separate items as far as most computer 
hardware is concerned. Combining the two on the same 
display requires special handling by the authoring 
software; not every system will do this. In fact, many 
authoring systems cannot produce or display piHel-based 
graphics at all. <Beware of the system which boasts "full 
•ttribute character graphic," as this is not graphics at 
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all but simply special tent characters. You cannotf -for 
examples draw a diagonal lin^ or an arc with such a 
sy E t©m m ) 

If the system supports graphics there are additional 
questions to ask ^ Does your training program require 
animation? and if so? does the authoring system support 
animation? Many systems tAjill only support static 
displays* and sometimes at great cost in terms of computer 
disk space* In order to produce the dynamic displays and 
simulations that are so crucial in nijclear training? the 
authoring system must provide animation. In the ideal 
case? the system will have the ability to do bothi import 
static displays? then allow you to enhance them in a 
variety of ways? including the addition of animation. 

In addition to text and graphics? nuclear training 
applications often require video. Many authoring systems 
will drive a videotape or videodisc player via the 
computer* Videodiscs have the advantage of relatively 
quick random access over videotape and therefore lend 
themselves better to interactive training* The 
disadvantage? however? is a significant increase in cost 
to produce such a computer—based program* 

There are several important considerations regarding 
the video— based authoring system? the most important being 
whether or not computer generated text and graphics can be 
displayed simultaneously with the video image. Generally? 
the electronic video signal and the computer tent and 
graphics signal are different types ( NTSC or composite 
versus RGB )* At the low end ? separate moni tors are 
required for each « Special monitors are now available 
which determine the nature of the incoming signal and 
display images accord ing 1 y^^but they may not show video 
and computer ten t /graph i cs s i mu 1 t aneous 1 y s 

Authoring systems which incorporate video and provide 
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the computer graph i c s / t en t overlay capability 
Csimul taneous display) are available* but be aware that 
hardware costs escalate for this capability. Special 
video and graphics adapter boards and monitors are 
necessary to convert signals and provide the overlay 
capability. These hardware costs start at several 
thousand dollars* 

The degree o^ friendliness of authoring software with 
regard to video management is extremely important. Some 
systems allow you to link with almost any video-driver 
software program and are therefore not tied to specific 
hardware configurations. This adds a degree of 
complexity, however * to program development. Other 
systems support specific hardware configurations and have 
many editing features built In. A caveat applies herei 
examine the requirements of the proposed training program 
before making a software decision. 

Table S summarizes authoring system features far 
visual display. 



Table a. Authoring System Support For Visual Displav 



Create? Edit? and Displ 



ay. 



Text TeKt over graphics 

Graphics Custom characters/fonts 

An i mat ion 

Import? edit? and display graphics 

Suppor t for . p . 

Videotape or di-ic 
Te>et /graph i c s overlay 
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Sound 

There are many training applications for which sound 
is important. At the low end an occasional "beep'' from 
the computer can provide useful feedback* At the opposite 
extreme? the reproduction of control room enunciators 
might be the best approach in some pr e=s i mu 1 a tor or 
refresher training^ 

At the very least the authoring system should allow 
you to program the computer's speaker for the simple 
applications. This is often done via ''music'' editors that 
are sometimes part of the graphics package^ Be aware that 
many systems do not have this capability? as simple as it 
may seem ^ 

There are many options available for reproducing 
recorded sound? and those options are growing rapidly* 
The simplest (but seldom used) method is to drive a tape 
recorder* Some authoring systems will allow you to link 
to software which will drive a recorder* An alternative 
is to use an audio track of a video tape or videodisc* 
There are problemss liowever? associated with providing 
sound over still images using this option^ Digitized 
sound is also an option? but it generally takes a 
relatively massive amount of disk space to stor. ihort 
segments of audio* 

Compact disc technology ( CD^-ROM and CDI ) is emerging 
as the solution to the storage space problem? but adopting 
it introduces new hardware and software options* At the 
time of this writing few authoring systems fully support 
campact disc technology^ If you must use it? it is wise 
to chnose hardware carefully and to select an authoring 
system that can easily import digitized audio files and 
play them at any time in a lesson* 
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Table 3 summarizes authoring system features 
associated with sound. 

Table 3, Autharing System Support For Sound 

Program the computer's speaker <*'beeps'' and music) 
Drive tape recorder^ 

Play audio track of video tape or disc 
Digitized sound 

Compact disc (CD-ROM or CDl) 



I n t er ac t i on 

Without meaningful interaction there is perhaps 
little need for using computers in training. While 
traditional computer-based training cannot reproduce the 
complex interactions that take place in training 
environments such as the classroom^ the simulator^ or on- 
the-job training^ there are clearly minimum interaction 
requirements for computer-based training programs to be 
meaningful. *'The computer" must have the ability to 
accept and process more than mere multiple choice 
responses if the safety mission of nuclear training is to 
be satisfied. Moreover, the computer must be able to 
provide feedback based on the judgment of a student*s 
response* In authoring system termss it must have the 
ability to provide soph i st i ca ted p conditional answer 
processing and branchingp 

It is at the level of answer judging and branching 
that language features should begin to appear in an 
authoring system. That iss some primitive rules of 
inference are required if interaction is to remotely 
resemble human interaction. This implies that there must 
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be some "if*** then capability built into the system . 

At the high end? this means an artificial intelligence 
inference engine! but this is not what most authoring 
systems support. Rather? a language is embedded i^ithin 
the authoring system which allows you to say? ''If the 
answer is close to ^source range detectors^ then judge it 
correctly and present frame #32 as feedback'* and the like. 
It should also have a simple but comprehensive means of 
specifying which answers are close to "source range 
detectors," For examples key word searches? spelling 
checks? wild cards? string parsing and concatenation 
should all be done without significant effort on the part 
of the author* There should be few or no restrictions on 
the number of paths to which to branch from a given 
interaction point* In addition? backward branching should 
also be possible* 

The judgment of interaction other than keyboard input 
must also be possible. At a minimum the authoring system 
must provide some sort of pointing device <via cursor 
keys? mousep light pen? etc.) and the ability to define 
"touch sensitive" areas of the screen* This provides for 
training in which a mechanical system^s components must be 
identified? for example* At a more sophisticated level? 
the authoring system should allow the trainee to 
manipulate portions of a graphic display to simulate the 
movement of switches and dials or the performance of 
maintenance operations in which assembly and disassembly 
is important* 

You will find that the ability of authoring systems 
to provide for interactive training varies from the barely 
capable to the most sophisticated* For most nuclear 
applications? those systems which approach the most 
sophist icated are preferred. 
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Table ^ summarizes authoring system features 
associated with interaction. 

Table A. Authoring System Suppart For Interaction 

Multiple input on a single screen 
Conditional answer judging 

Key word search, spelling checkt wild cards 
Forward L bac kward branch i ng 
Cond 5 t iona 1 branch i ng 

Pointing device<s) & " touch=sensi t i ve " screen areas 
Manipulate screen areas 



Rea 1 i sm 

Realism in the nuclear training environment means , at 
the least, the use of actual plant operating parameters. 
Whether the training application is on a routine plant- 
related calculation or involves complex simulation, real 
information must be used. Incorporating such realism into 
computer-based instruction should require the least amount 
of by-hand data collection and program management. That 
is, computers are used to create and maintain databases of 
many classes of information. Plant operating parameters, 
parts inventory, operating history^ emergency procedures? 
and many other bits of information are routinely stored in 
databases* The authoring system should be the vehicle by 
which this information is tapped and kept up to date in 
training programs* 

What this means is that the authoring system must 
have the ability to access external databaseSf select 
relevant information from them, and use this informatian 
in lesson material. While this process may seem rather 
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demanding^ it can be broken into simple steps; searching 
da t abases » wr i t i ng se lec ted i n^orma t ion to fil eB f r ead i ng 
external filers and assigning information found in the 
files to authoring system variables« If the authoring 
system and database system in combination can perform 
these steps? then the process is possible- The process 
guarantees that training programs are realistic and up-to- 
date (at least as far as the databases are up-^ to— da te ) , 
Moreover 5 it removes a level of error associated with 
human retrieval of realistic data and incorporating it 
into training material by hand- 

In the category of realisms as defined aboves there 
are few authoring systems which provide such 
sophistication. In fact? most authoring systems do not 
provide user-definable variables and therefore cannot even 
pass data from frame to framej let alone from external 
application to the lesson and vice— versa. Some systems 
will execute external app 1 icat ions ? b'lt do not have the 
ability to pass data back and forth or read and write data 
f i les. 

Nuclear trainers should carefully consider realism in 
their training programs* It is not a simple matter to 
search out the required features within authoring systemsl 
it is perhaps even more difficult to find a working 
demo nst rat ion of such features - If you are tasked with 
authoring systems selection, it is well worth the time and 
effort to determine which system(s) contain the features 
which support realism. 

Table 5 summarizes authoring system features 
associated with realism. 
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Table 5, Authoring System Support For Realism 



Author— def 1 nab 1 e variables 
Access eKternal databases 

Pass data between lessons and databases 
Read and write data "files 



SUMMARV AND CONCLUSION 

The mission o-f nuclear training places stringent 
standards on the content and design of its programs^ 
Computers provide an attractive means of presenting 
instruction^ but must meet the rigorous standards imposed 
on all presentation media. As such s the tools for 
creating computer-based instruction must be carefully 
scrutinized for their capabilities to produce training 
material which satisfies the standards- 
Authoring systems for computer —based instruction are 
numerous and diverse in their capabilites. Many do not 
support features capable of creating instruction to 
nuclear field standards. It is incumbent upon trainers in 
this field to carefully e><amine both the content and 
instructional design of programs they wish to deliver via 
computers and then embark on the authoring system 
evaluation and selection process. This is different than 
selecting the "best" authoring system by comparing 
authoring systems. An authoring system is not a single 
tool but a collection of tools! it is only after the job 
is understood that the right collection of tools can be 
selected . 

A list of articles and publications related to the 
authoring systems evaluation process is attached to assist 
those who are facing the process^ 
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COMPUTER BAg ZD TRAINING FOR NPP PERSONNEL 
(INTERACTIVE COMMUNICATION SYSTEMS 



AND FUNCTIONAL TRAINERS ) 



H . . Mart in 



ABSTRACT 

K W U as a manufacturer of thermal and nuclear power 
plants has extensive customer training obligations within 
its power plant contracts. In this respect KWU has gained 
large experience in training of personnel, in the product- 
ion of training material including video tapes and in the 
design of simulators* 

KWU developed interactive communication systems (ICS) 
for training and retraining purposes with a personnel com- 
puter operating a video disc player on which video in- 
struction is stored. The training program is edited with 
the help of a self developed editing system which enables 
the author to easily enter his instructions into the 
computer- Another special application is designed for a 
motoric repetition and testing of interlocks in complex 
systems s where the operating staff has to recheck its 
knowledge in short intervals. Further mathematical model- 
ling of system behaviour is performed through simple simu^ 
lation procedures in order to intensify the interaction 
between the trainee and the training system, ICS enables 
the plant management to better monitor the performance of 
its personnel through computerised training results and 
helps to save training manpower* 

German NPPs differ very much from other designs with 
respect to a more complex and integrated reactor control 
system and an additional reactor limitation system* Simu- 
lators for such plants therefore have also to simulate 
these systems. KWU developed a "Functional Trainer" (FT) 
which is a replica of the primary system, the auxiliary 
systems linked to it and the associated control* limita- 
tion and protection systems including the influences of 
the turbine operation and control. It calls for approxi- 
mately 30 % of the investment cost and can cover almost 
80 % of the training program of a full scope simulator. 

Another approach is the "Nuclear Plant AnalyEer" 
(NPA) which provides graphical displays and systems status 
on coloured CRTs coming from design codes run on a main- 
frame computer via telephone line. 



377 



V.A.7.2 



KWU*S TRAINING PHILOSOPHY AND STRUCTURE 

K W U'S function as a manufacturer of nuclear power 
plants in the Federal Republic of Germany differs signi- 
ficantly from that of foreign manufacturers. This is main- 
ly due to the fact that KWU is a turnkey contractor. From 
the initial fuel loading of the reactor until handover to 
the operating company, KWU is a joint applicant for and 
holder of the licence under the terms of the West German 
Atomic Energy Act. In addition, KWU is responsible for the 
plant-oriented training of the customer's staff and provi- 
des a large variety of training jervices for German and 
foreign customers. 

KWU developed a modular training system with which it 
is possible to take into account the differing require- 
ments of various projects and customers^ without having to 
work out a new program each time. The essential parts of 
the training system are basic training and specialist 
training. Basic training comprises basic technical courses 
up to instructor courses. 

Specialist training encompasses six groups of train- 
ing modules. The first group consists of technical system 
and plant courses^ the second group is similarly 
structured^ and provides simulator courses. All the theo-- 
retical courses for specialists are included in the third 
group; the fourth and fifth groups cover practical 
on-the--job courses. And the sixth and last group comprises 
the phase of familiarization with the customer's own plant 
during erection and commissioning. 

The training program for NPPs in the FRG can be di- 
vided into five steps (Fig. I) 

- The preparatory phase with basic theoretical training 

- The practical training on-the-job in similar install 
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lac ions 

- The plant relaced special Craining 
^ The simulator training 

- The commissioning of the own plant. 

The basic theoretical training consists normally of a 
3 months course in a research center, at a technical uni-- 
versity or in a utility owned school and a 6 weeks systems 
introductory course at KWU . Practical training is crga-^ 
nized by tb 3 utilities themselves while the plant related 
special training is provided by KUU . 

Before ICWU offers the customer a training program for 
the various groups of personnel^ an optimum organization 
of the plant, with job descriptions and job specifica^ 
tions, is proposed. This is of particular interest to for- 
eign customers. The personnel to be trained is categoriEed 
by qualifications. By means of this procedure and with the 
aid of the modular training system, standardized training 
courses can be compiled from the individual training 
mod u 1 e s * 

Each participant of a theoretical course receives ac- 
companying training material for each topic with text, da- 
ta and graphics. Lectures are held in the morning; the af- 
ternoons are reserved for seminars. At the end, the parti- 
cipants in the seminars present their results before the 
wh ole class. 

At the end of each subject section, a colloquium is 
held, in which an expert answers questio:is that are still 
open, and gives a final survey. At regiaar intervals, 
written tests are given; part of them consist of multiple- 
choice questions, which can be evaluated quickly with the 
help of a minicomputer* At the end of a training session, 
which does not normally last longer than two months, a 
formal intermediate eKamination is held* 
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The training programs developed by KWU are part of 
the programs which West German operators must submit to 
the regulatory agency in order to obtain an operating li-- 
cence. The intermediate examinations of the plant courses 
are approved by the authority as written parts of the li- 
censing exam. In part, representatives of authorities ob- 
serve or participate in K W Lf courses. 

Simulator training for German utility personnel nor^ 

mally is within the scope of the utility run simulator 
training center. However^ since 1983 KWU is heavily invol-- 

ved both in simulator initial and retraining activities 

for the German preconvoy and convoy plants. 

For all this K W U disposes of a customer training or-- 

ganizacion with studio facilities for professional video 

and film production, the manufacture of training materia'. 

and specialists for the design of training programs and 

individual courses as well as the instructors for the va^ 

rious special subjects. 



INTERACTIVE COMMUNICATION SYSTEMS 

For the purpose of individual training and retraining 
KWU developed an interactive communication system (ICS5 
Fig. 2). 

ICS is a media where the flow of information is no 
one way street from the media to the trainee. The success 
of the trainee in digesting the information can easily be 
monitored and thus is kept evident compared to other 
me d i a . 

ICS consists of a training course^ structured with 
the help of a table of contents and various types of test 
questions after each section with additional instructions 
or b ranch ing . 
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The crainee can start any section chrough the table 
of contents or in defining a subject word. In this case, 
the computer starts the program on video disc at this sec- 
tion whei.-e the subject word is explained. The final test 
is randomly organized so that never the same test sequence 
is provided. The trainee gets an immediate feedback on his 
performance after each question. He can delete individual 
questions or he can decide to answer them later. At the 
end he gets his personal result and an indication of those 
sections of the lesson where he gave wrong answers. 

For those subjects where the real working environment 
has to be shown, i. e. repair manipulations etc. or where 
complex technological phenomenon can only be displayed by 
animation a video disc is used which is linked to the PC 
Via an especially developed video mixing equipment. In the 
case of repetition of logical conditions of plant systems 
and procedures as part of the retraining program only a 
motoric test is necessary. This is performed by the PC it^ 
self without requiring video instructions, the PC genera^ 
ting the P&ID directly on screen. 

An editor software package was developed which re- 
quires no programming knowledge and is easy to use. The 
user can design training courses, divided into sections by 
himself following the program menu. After the sections he 
integrates test questions for which the editor provides 
various types of question formats. 

Text overlays to video images are possible as well as 
area assignment or user assignments on screen. Modifica-^ 
tion of inputs are easy to make as well as the addressing 
of images. No time code conversion is required. 

Also small simulation applications are included in 
the system in order to make the training more practical 
oriented . 
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The video material is produced in the KWU owned stu^ 
dio, mastering is done outside. The hardware used consists 
of a Siemens PC compatible with IBM standard, a sCandard 
video disc player and the electronic video niixing equip-- 
ment. If a utility wishes to document the training result 
or individual screens a printer can be added. All this 
equipment is integrated in the KWU Teach Tower. 



NUCLEAR PLANT ANALYZER 

The KWU "Nuclear Plant Analyzer" (NPA) is a real time 
engineering simulator based on the KWU computer programs 
used in plant transient analysis and licensing. 

The r^-imary objective is to promote thy understanding 
of complex technical and physical processes of a NPP du- 
ring malfunctions or malfunction combinations. 

This has been achieved by the application of the 
transient code NLOOP and operator interaction including 
all simulator functions normally available in training 
simulators. The mainframe computer in the KWU computing 
center drives four colour graphic displays controlled by a 
dedicated graphic computer by telecommunication via te^ 
1 e ph one* 

No control room is available since emphasis is put on 
the graphical display of variables and system status 
(Fig 3) . 

Three types of pictures are provided: 

- semigraphic pictures (symbolic plant diagrams e, g.) 
with process data as alphanumeric datas analog bars (wa^ 
ter level e. g.) and graphic symb ola (pressurised 
spraying e * g . ) 

- curve pictures containing up to 6 axes with up to 6 cur^ 
ves each (trend curves for power, mass flows etc*) 
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- lists with a 1 ph a^nume r i c data ( annun c i a C i o n s , operator 
interactions etc.) 

Some pictures conCain fields where user defined va- 
riablea can be displayed. The assignment of a specific 
piccure to a display is at the liberty of the operator. 

One version of the NPA based on the code NLOOP is in 
operation for KWtJ 1300 M^W Standard PWR plants. It can be 
adapted to suit other P^WR plants by some r e p r o g r amm ing to 
meet the individual demands especially in the field of 
control and instrumentation. 

The objective of the NPA is to be used for training 
of higher level, managerial NPP personnel, not directly 
involved in shift operation. Another application is to be 
used for transient analysis giving a prompt feedback even 
if parameters are changed. Thus it can also be used as a 
means of malfunctions logistics and to assist the technic 
cal support center in case of emergencies. 



FUNCTIONAL TRAINER 

During 5500 h of training of shift personnel on a 
full scope simulator both for initial training and re^ 
training purposes KWU has run an evaluation program about 
the amount of utilisation of the different areas of the 
simulator control room. The analysis lead to the conclu- 
sion that up to 80 1 of the total training can be provided 
by representing only 30 % of the full scope control room. 

This is mainly due to the fact that German NPPs in 
addition to other designs have a special reactor limita- 
tion system separate from the reactor control system and 
the protection system. It is designed on the basis of the 
same safety standards as the protection system, i. e. 
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4 channels and initiates automatic actions to locally and 
globally porotect the core and the primary syatem in case 
of tranaieramts before the protection systems is activated. 

The tr^aining of these complex systems therefore plays 
an Importa^^ t part in the overall simulator training pro* 
gram althoi_agh it requires only a part of the simulator. 
The design of this "Functional Trainer" (FT) calls for the 
detailad s imulation and representation of the primary 
aygtem, th^ e auxiliary systems directly connected like 
emergency ^ore cooling, volume control or extra borating 
system. The = ^secondary system is simulated as a closed cy* 
cle x^ith d^Etailed representation of life steam, feedwater 
supply and - emergency feed water. 

Other systems are represented by replacement simula- 
tion. Redun dant trains, i. e, ECCS are totally simulated, 
however^ on ly I of 4 trains is represented in the control 
rooin with a dditional position switches for the indication 
of the Qthe parallel trains. 

The FT also contains a high resolution computer aided 
procass inf=»rmation system (PRINS) for the display of mul^ 
t i f un c t ionaD. diagrams, trend indications or system status 
displays (f-±g. 4). The FT uses two SEL computers of the 
32 series which are connected with the control room 
interface v^S^a fiber optics* 

The FT is mainly used for practical exercises and fa^ 
miliarizaticon with the reactor control systems, limitation 
systenis, tu:^bine controls and reactor protection system. 
This requir. ea a scope of simulation which allows load 
follow operajtion, 3 loop operation, hot start-up after re^ 
accor or tu^^rbine trip, load rejection to inplant supply 
and total st art^up and shut-down. 

It goeB without saying that the FT can also be used 
for deeign- ob j ec t i ve- o r i en t e d strategy and behaviour 
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training . 

The instructor can ^program eKercises in paralle 1 with 
tha training session by dafiniiig initial conditions ^ mal-- 
functiona or degradat iof» s , trend recordar; or CRT d dsplay 
parameters a^s^o^ 

The FT is also use*^ for tha further developinsnt of 
the computer aided proca mm information s y s t a in ( PR IN ^ ) for 
new plants. If installed at site the FT can bo linfeLed in 
parallel to the real plar^t via the data acquisition compu^ 
ters which normally fee-<3 Chie PRINS process ing coni^uter. 
Thus it cari be used as an "On-^line Predictive Analyzing 
and Learning System^' (Op^s^L), 
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COMPARING INTERACTIVE VIDEODISC TRAINING 
EFFECTIVENESS TO TBADITIONAL TRAINING METHODS 



Nancy Kenworthy 



ABSTRACT 



Videodlec skills training programs devsloped by 
Industrial Training Corporation are being used 
and evaluated by major industrial facilities* In 
one euch study. Interactive videodisc training 
programs were compared to videotape and 
Instructor-based training to determine the 
effectiveness of videodisc in tarms of 
performance J training time and trainee 
attitudes* Results showed that when Initial 
training was done using the Interactive videodisc 
system^ trainee performance was superior to the 
performance of trainees using videotape , and 
approximately equal to the performance of those 
trained by an Instructor* When each method was 
used in follow-up training , interactive videodisc 
was definitely the most effective* Results also 
indicate that training time can be reduced using 
Interactive videodisc. Attitudes of both 
trainees and instructors toward the interactive 
videodisc training were positive* 



DESCRIPTION OF THE STUDY 



This evaluation methodology was designed and 
carried out by the training staff at a major 
Industrial facility * The purpose of the study 
was to evaluate the effectiveness of Interactive 
videodisc training compared to videotape and 
Ins true tor--based lessons * 

In this study ^ journeymen mechanics with little 
or no formal training in electrical/electronic 
skills were trained in two electrical skills 
areas: using an oscilloscope and using a 
multimeter. Participants in the study were 
volunteers * 
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it-??ITIAL lUAINING EVALUATION 



Bifore the tfc-ainlngj instructors at the facility 
developed tr^ainlng objectives in the two content 
argas * Froni these objectives ^ a written pretest 
post-test J at^d performance tasks were developed 
for each corit=ent area. All trainees were given 
the written ^^retest and were evaluated on the 
performance t=asks . The pretests were left 
ungraded , locked away until the conclusion oj 

training in Oorder to avoid biasing the results oi 
the study . 

Trainees wetra^ randomly divided into three 
groups , Tral — nees in each group received Initial 
training in t rlie content area through a specific 
aethod Int :eractlve videodisc, videotape^ or 
Instructor^ba^^sed classroom training. 

The trainees in the interactive videodisc group 
received tral^ning from two interactive videodisc 
liisons devel *oped by Industrial Training 
Corporation* Both lessons are Level III 
intaractlve vrideodlsc training programs* These 
off-the-shelf lessons consist of a vldeodisCj the 
neeissary com^puter software ^ a user's reference 
containing th^e lesson objectives, and a user's 
handbook whic^ti provides the information necessary 
to operate th^e hardware delivery system. Before 
"beginning the trainings each of the trainees 
using the vld^Bodisc system received a five-minute 
eKplanatlon o^ff how to use the system. After this 
initial explaE^iatlon , trainees were permitted to 
ask questions about system operation if 
necessary, Hc^waver, the Instructor was not 
perinltted to ^answer questions about lesson 
content. 

Trainees in tt^e videotape training group used 
off-the-shelf videotape training programs in the 
content areas These videotape programs were 
selected by tt^e training staff at the facility. 
Trainees had fc^he videotape and accompanying 
workbook, Tr^»lning was done on an individual 
basie so that each trainee could review material 
as much as nec^essary within the time limit. 
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Trainees in the instructor-based training group 
received training through a classroom lectura 
format with demonstrations. Training was done ^n 
small groups* Average trainer-trainee ratio was 
1:3, The instructors were experienced in the 
content area and had taught these subjects 
previously* Instructors were permitted to 
provide demonstrations with actual equipmantj and 
use handouts and the chalkboard as teaching aids * 

Each group was given a time limit of two hours to 
complete each lesson. All groups used the intl^e 
two hours , 

After completing the lesson, trainees were 
administered the written post-test and 
performance tasks. The effectiveness o£ thi 
training for each method was baaed on the 
improvement shown by comparing pretest and 
post-test scores and performance task 
evaluations , 



PERFORMANCE INITIM. TRAINING 



By comparing the mean pretest and mean post^'teat 
for each group, it is possible to determine how 
much each group improved as a result of the 
particular training method* Written tests were 
scored as a percentage of correct answers. 
Performance evaluation scores reflect the 
percentage of tasks performed according to 
specified criteria * After initial trainings the 
Improvement resulting from interactive videodlac 
training was comparable to the improvement 
resulting from the instructor-based training 
both on the written test and on the performance 
task evaluation* The improvement after vldiOtflp« 

instruction was substantially lower. The 
improvement percentages were consistent for botH 
the oscilloscope lesson and the multimeter 
lesson , 
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Initial Training 



Mean Sc&»res OBcilloseope 

yidegdls-- c Written Per^ Jormance 

PritiSt 29% 0% 

Post-tes t 48% 47% 

Ifflprovens^erit 19% 47% 

Priteflt 30% 0% 

Post-tei t 40% 27% 



fem*^nt 10% 27% 

Pretift 33% 0% 

Post-tei— t 50% 53% 



reBi«nt 17% 53% 

Mean Sen-— rg.g -- Multimeter 

YUmdiamc Written Peyjgormance 

P«test 47% 7% 

Post-testt 66% 53% 

Inproveff«nt 19% 45% 



PtetBst 40% 13% 

Post-testt 50% 47% 

eroent 10% 34% 




Pretiit 32% 0% 

Post-teg^ 50% 53% 

Irnproveni^nt 18% 53% 
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FOLLOW-UP TRAINING EVALUATION 

After completing the Initial training lesson^ 
each group was scheduled for follow-up training 
using the other two methods of instruction tested 
in this study* The purpose of this follow-up 
training was twofold: 

- to give each trainee an exposure to each 
method so that trainee attitudes toward 
each method could be evaluated 

to provide some indication of the 
effectiveness of each method as follow-up 
training. 

At the conclusion of each lesson 5 trainees were 
again administered the written post-test and 
performance task evaluation so that any 
improvement could be: noted. 



PERFOR^mNCE FOLLOW-UP TRAINING 



Here^ some surprising results were noted. After 
completing the interactive videodisc lesson , 
trainee scores improved a Ignif Icantly ^ no matter 
whether they received the videodisc training 
firsts seconds or third in the rotation. 
Performance improvement using interactive 
videodisc for follow-up was superior to both 
instructor-based and videotape methods , These 
results were consistent for both the oscilloscope 
lesson and the multimeter lesson * 

The Interactive videodisc seemed especially 
effective in Improving scores on the performance 
tasks after Initial training * The trainees 
improved their performance scores an average of 
more than 30% following the videodisc training. 

Since the follow-up training results were a 
corrollary to the primary study of Initial 
training evaluation 5 they provide food for 
thought rather than conclusive results. 
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This is an area that ITC intends to pursue in 
future studies . One possible explanation for the 
dramatic Increase In performance scores may be 
that ITC's videodisc skills training lessons are 
designed around simulations fo hands-on tasks 
that provide practice for the trainees • 



Improvement After Initial Training 

Oscilloscope Written Performance 

Videodisc 11% 30| 

Videotape 2% 41 

Instructor 9% 201 

Multimeter 

Videodisc 13% 36% 

Videotape 4% 1% 

Instructor 8% 27X 

TRAINING TIME 



Each lesson was limited to two hours in order to 
provide a fair basis for comparing performance* 
However J surveys of trainees and instructors 
after training indicated that interactive 
videodisc may provide a way to reduce training 
t ime . 

Trainers in the instructor-based training 
complained of a lack of time to adequately cover 
the material. Trainees in the instructor-based 
groups described the speed of presentation as the 
primary weakness of this method of training. 
Trainees in the interactive videodisc training 
groups felt they had adequate time to complete 
each lesson. 
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TRAINEE /TRAINER ATTITUDES 



After completing the lessons in each of the three 
training methods ^ trainees were surveyed to 
determine their attitudes toward each training 
method. Trainees were encouraged to comment on 
the strengths and weaknesses of each method j and 
to suggest ways that the training might be 
improved , Instructor comments were also 
solicited , 

Trainee attitudes paralleled performance in terms 
of which methods were most effective* When asked 
what method of instruction they thought worked 
bests trainees were fairly evenly divided between 
interactive videodisc and Instructor-based 
training. However, when the question was 
rephrased to ask which training method they would 
prefer if only one method were available , a clear 
majority chose videodisc. When asked why^ 
trainees gave two reasons: 

The instructor often went too fast* 

The interactive videodisc protected 
trainees from the embarrassment of giving 
an incorrect answer In front of other 
people * 

Trainers were positive about the flexibility of 
interactive videodisc in terms of where and when 
it can be used for training. They also approved 
o£ its use for Independent training. The 
Instructors compared the effectiveness of 
interactive videodisc to their best Instructor. 

Trainees using the interactive videodisc made one 
suggestion for Increasing the effectiveness of 
the training that woi:ld be simple to implement. 
In this case 9 since the training on the videodisc 
was generic in nature ^ trainees thought it would 
be helpful to have the specific equipment that 
they would be using on the job available for 
practice after CQmpleting the simulations on the 
videodisc * 
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METHODS FOR EVALUATION OF 
INDUSTRY TRAINING PROGRAMS 

Doldres S. Morisseau 

Mary Louise Roe 
Jul [us J, Persensky 



ABSTRACT 

The NRC Policy Statement on Training and Qualification 
endorses the INPO-managed Training Accreditation Program 
in that It encompasses the elements of effective 
performance-based training. Those elements are: 
analysis of the Job, performance-based learning 
objectives, training design and implementation, trainee 
evaluations and program evaluation* 

As part of the NRC independent evaluation of utilities' 
implementation of training improvement programs ^ the 
staff developed training review criteria and procedures 
that address all five elements of effective performance- 
based training* The staff uses these criteria to perform 
reviews of utility training programs that have already 
received accreditation. Although no performance-based 
training program can be said to be complete unless all 
five elements are In place, the last two, trainee and 
program evaluation, are perhaps the most important 
because they determine how well the first three elements 
have been implemented and ensure the dynamic nature of 
training* 

This paper discusses the evaluation elements of the NRC 
training review criteria* The discussion will detail the 
elements of evaluation methods and techniques that the 
staff expects to find as integral parts of performance- 
based training programs at accredited utilities* 
Further, the review of the effectiveness of 
implementation of the evaluation methods is discussed* 
The paper also addresses some of the qualitative 
differences between what 1s minimally acceptable and what 
Is most desirable with respect to trainee and program 
evaluation mechanisms and their implementation. 
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EACKGROUND 

On March 20, 1985, the NRC Issued a Final Policy Statement on Training 
and Qualification of Nuclear Power Plant Personnel, This guidance 
endorsed the INPO-Managed Training Accreditation Program because the 
NRC believed that the Accreditation Program encompasses the elements 
of effective performanced-based training. These elements are* 
analyses of the job, performanced-based learning objectives, training 
design and implementation , trainee evaluation, and program evaluation. 
Additionally, the NRC developed NUREG-1220, "Training Review Criteria 
and Procedures," to assist the NRC staff in performing independent 
evaluations of industry training Improvement programs including post- 
accreditation audits. The evaluation criteria and accompanying 
procedures provide detailed questions for determining the extent of 
implementation of the five elements of effective performanced-based 
training* While all the elements are essential for effective 
training, perhaps the last two, trainee and program evaluation, are 
more Important because they determine how well the first three are 
Implemented and are the basis for continuing Improvement of training. 
The focus of this paper will, therefore ^ be concentrated on evaluation 
methodology including what the NRC staff expects to find as the 
integral part of performanced-based training evaluation at accredited 
utilities, and how effectively those mechanisms are implemented. The 
paper also addresses some of the qualitative differences between what 
is minimally acceptable and what is most desirable with respect to 
trainee and program evaluations. 

The trainee evaluation process includes an assessment of the 
individual prior to entry into the training program, evaluations 
during training, and evaluations while the individual is on the job. 
Aspects of th.u assessment includes exemptions from training based on 
experience, prior training or performance-based testing; regular 
job-related evaluations with feedback; and remediation. The 
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focus of program evaluation is to determine the training program's 
effectiveness as it is implemented, and to revise It as necessary. 
Aspects would include aggregate evaluation of trainee test 
performance, trainee and instructor critiques, on-the»job evaluations 
by trainees, feedback from supervisors, formal program evaluation, and 
evaluation of training staff performance. The NRC staff performing an 
audit of industry training programs reviews training procedures , 
records, and tests and also interviews the training staff, trainees 
and job supervisors to determine the effectiveness of trainee and 
program evaluation, A more detailed discussion of evaluation 
methodologies is presented in the remainder of this paper. 

TRAINEE EVALUATION 

Exemptions 

The first question included under the element of trainee evaluation Is 
that of exemptions from any portion of training* A determination is 
made as to the circumstances under which exeinptlons from training may 
be granted. Minimal standards would allow exemptions without any 
auditable record, A more acceptable critsrion for exemptions would be 
documented evaluation of equivalent training or performance-based 
testing. The most desirable method for granting exemption from 
training would depend exclusively on performance-based testings 1,e,, 
testing based on the knowledges and skills needed to perform the 
specific job. 

Test Items 

Trainee evaluation should be appropriate to job performance 
requirements and training objectives. Test items for tasks selected 
are reviewed to determine that test items for all tasks are consistent 
with the learning objectives and job performance requirements. The 
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most appropriata test items would require faithrul Indication of 
required actions given the stated conditions to a predetermined 
standard of performance, so that knowledges and skills are determined 
to be adequate. 

Feedback 

The trainees should be provided prompt^ regular^ objective feedback on 
their performance. Examples include the self-checks which can be 
scored by the trainee, as in self-study workbooks, or Instructor- 
administered and scored tests. In the latter case, tests should be 
scored and returned within a day, but 1n no more than a week. In 
addition to feedback provided for the overall performance, feedback 
should also be provided relating to specific skills and knowledges 
that do not meet job performance requirements (1,e,, learning 
objective standards). 

Performance Below Minimum Standards 

Trainees who perform below minimum standards should be provided 
remedial training, retested and removed from the training program if 
minimum standards are not met. When difficulties occur in original 
training, special training techniques should be used to correct thm. 
This would require careful analysis with Input from the trainee. 
Information regarding trainee deficiencies should also be used as 
feedback into the program. Job incumbents performing below minimum 
standards during requal Iflcation or continuing training should be 
removed from associated Job duties and provided with remedial 
training. Root causes of failures In continuing training programs 
should also be identified and used to Improve that phase of the 
program. During a post-accreditation audit, the staff reviews 
specific examples of the remediation process including additional 
training provided, procedures for retesting and procedures for removal 
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from the program or job duties for failure to meet the minimum 
standards. 

Compromise of Test Contents 

Training procedures should define the steps that are taken to preclude 
compromise of test contents. For instance^ a sufficient number of 
questions In the test bank facilitates variations 1n examinations so 
that questions need not be repeated between tests* A single 
administration of a particular examination (or several versions per 
administration) 5 regular examination proctorlngs and effective 
procedures for and Implementation of measures taken against exam 
compromise are other measures reviewed by the staff, 

PROGRAM EVALUATION 

The audit team looks for evidence that the utility has a systematic 
method in place not only to evaluate the training program's 
effectiveness s but also to revise the program when necessary. There 
are a number of mechanisms that are good parameters for program 
evaluatlonp e,g*s evaluation of test resultSs programnatlc critlquesp 
on-the-job feedback^ formal evaluations and evaluation of training 
staff performance. 

Evaluation of Test Results 

IdentiflCfjition of trainees* weaknesses Is relatively straightforward. 
Finding the cause of those weaknesses is not as simple and more 
training is not always the answer. One method of Identifying 
weaknesses is evaluation of aggregate results of both written and 
operating tests to improve testing and to provide feedback to improve 
training, A good evaluation of test results usually Includes a test 
item analysis. Test items that are consistently answered Incorrectly 
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may pinpoint the cause of program weakness and narrow the decision as 
to whether the fault may lie with the adequacy of procedures or other 
job performance aids provided, selection criteria, sequence of 
training or the qual if ications and performance of the instructor. 
Evidence that such an analysis 1s implemented is usually found in 
procedures or an actual evaluation of test results from a recently 
completed course. 

Instructor and Trainee Critiques 

Both trainees and instructors are In unique positions to provide 
valuable input to the program eva^^uatlon process. Instructors can 
identify problems with technUal accuracy, CQmpleteness , sequencing, 
and trainees* difficulties with training materials. Trainees can 
identify the same type of problems from the perspective of the target 
population. As part of a post-accreditation review, the staff 
determines If there 1s evldenca that instructor critiques of training 
are required, are an ongoing part of the conduct of courses, and, most 
Importantly, that there is a formal method for the timely 
implementation of appropriate changes to the program. The staff also 
reviews requirements for trainee critiques and ensures, through 
documentation, that they are structured, focused on desired 
information, and used for program evaluation* 

On-the-Job Experience 

A formal program for collecting on-the-job experience Information from 
job incumbents 3 to 6 months after they complete training should be 
Implemented and the data used for program evaluation of Initial 
training. Continuing training can also be evaluated using similar 
experience Information from job incumbents. The information collected 
should include: 
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(1) unexpected difficulties 1n performing tasks on the job, 

(2) those tasks that are particularly difficult or easy to 
nerformp 

(3) any difference between the way tasks are performed on the 
Job and the way they are taught, 

(4) additional training needed to do the jobp and 

(5) kinds of errors committed on the job. 

The staff reviews documsntation to determine the existence of a formal 
program to collect on-the-job experience in the five areas above and 
to determine if the information Is part of the evaluation and 
modification of initial and continuing training programs* 

Supervisor Feedback 

A formal program to periodically solicit supervisor feedback on Job 
performance problems is an Integral part of program evaluation and the 
Information obtained should be used to Improve both initial and 
continuing training* Information collected from supervisors should 
Include (1) tasks for which job incumbents are inadequately prepared, 
(2) kinds of errors committed by job incumbents, (3) additional 
training received by new Job Incumbents once they are on the job, 
(4) suggestions for Improvements in Initial and continuing training 
programs, and (5) expected changes in job assignments, procedures, or 
equipment* The NRC reviews evidence to ensure that this feedback Is 
procedural Ized as a part of the program evaluation process and that 
the information is actually used to evaluate and Improve Initial and 
continuing training* 
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Internal and External Program Evaluation 

The NRC audit determines 1f there have been Internal or external 
program evaluations for the two most recent years and determines if 
the findings have been Incorporated In the program. If the findings 
are not useds there should be evidence of appropriate rebuttal of the 
findings* External evaluations can be either from outside of the 
utility or merely external to the training organization, e.g., utility 
performed QA audit. 

Training Staff Performance Evaluation 

Training procedures should be in place that require a periodic (once 
every 12 months at a minimum) evaluation of the performance of 
training staff members. The NRC reviews to ensure that there Is 
evidence that these procedures are being Implemented 1n a consistent^ 
accountable manner. 

CONCLUSION 

Without trainee and program evaluation elements , performance-based 
training would not have true substance or be effectively modified. 
For training to be truly performance-based it must be dynamici without 
evaluation and feedback any training program will become static* 

Information derived from the evaluation process must be used to 
determine program strengths and weaknesses which are then used as 
feedback to further enhance the training program's content and 
effectiveness. The review criteria of NUREG-1220 used by the NRC are 
designed to determine 1f utility training programs Include consistent 
evaluation methodology with evidence of auditable documentation. 
Appropriate implementation of these two elements will ensure the 
dynamic nature of industry training. 
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ACCREDI TATION S ELF-EVALUATION I AN EFFECTIVE 
PROGRAM EVALUATION TOOL 

Ronald L. Fritchley 



ABSTRACT 

The Institute of Nuclear Power Operation's (INPO) 
Accreditation Program inGludas a systematic 
evaluation that subsequently improves nuclear 
utility training programs. The process begins with 
a utility-conducted self -evaluation that measures 
its training programs against the accreditation 
criteria and objectives. When properly 
conducted^ the self-evaluation results should 
Identify weaknesses within each program as well as 
program strengths, utilities are then expected to 
take the necessary actions to correct the 
weaknesses that have been found. Experience with 
the process has shown that a properly conducted 
self ^evaluation can also be an effective program 
evaluation. In addition^ much of the data and 
information that are collected and used during the 
self-evaluation process are also used in 
conducting other evaluations of the training 
organization and programs. This paper will 
discuss using the seif-'evaluatlon process as a 
tool for conducting program evaluations. 



INTRODUCTION 

At the last Symposium^ INPO staff provided an update c 
the status of accreditation in the nuclear power industry. 
At that time half of the utilities had filed an acceptable 
self-devaluation report for at least one training program ar 
slightly more than 3 0 programs had achieved accreditation. 
On December 16, 1986, the nuclear power industry achieved a 
significant goal by having 10 primary training programs at 
61 specific operator plants accredited or ready for 
accreditation with over half of these 610 programs having 
achieved accreditation* 
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During the first few years of implementing the INPO 
accreditation program^ all of us have learned a great deal 
about the process and how it can be used to improve traininc 
programs. My discussion today is based to a large extent on 
observations of the self-evaluation proce'^s and how it has 
been conducted by several nuclear utility training 
organizations. The following is a description of how the 
accreditation self ^evaluation and program evaluation 
processes can be combined to make both of them more 
effective and efficient. This paper primarily addresses 
program evaluation as a measure of the effectiveness of the 
training process and the proper conduct of the training 
program* It does not include evaluating the results of 
training by assessing job performance. 

SELF-EVALUATION 

The self "^evaluation process as it has evolved in the 
INPO Accreditation Program has three principal purposes«= it 
is intended to help the training organization and its 
programs improve; it provides a framework for action plans 
and resource requirement estimates; and it provides a firm 
foundation for the preparation of the accreditation 
^elf ^evaluation report to be submitted to INPO, The 
self--evaluation is the most important part of the entire 
accreditation process. 

The task of conducting and managing a self ^evaluation 
requires a team effort to produce the desired results. 
Therefore, the process must be planned in detail and 
conducted in an efficient manner. In addition, it is 
desirable for a self- Bvaluation to have the following 
attributes i 

o The process should be internally motivated. The 
literature on self ^evaluation and organizational 
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effectiveness suggest that an internally motivated 
self ^evaluation is far more suGcessful than one the 
is conducted with the single purpose of responding 
to an external agency. 

o The plant and utility management is Gom.mitted to the 
process. Management can demonstrate this commitment 
through goals and objectives, their advice 
and recommendations on key training issues and 
making resources available to take corrective 
actions , 

o The design of the self-evaluation should be 

appropriate to the utility, plant, and training 
organization* This should include participation of 
personnel that represent the various organizational 
units including plant departments involved with 
training. 

o The process should be well led. Effective group 
process^ problem clarif icatic a and solving^ and 
group leadership skills must be used, 

o The participants should be committed to a thorough 
and critical self-devaluation* 

The properly conducted self-evaluation is a thorough 
and critical comparison of a utility's training programs no 
only to the INPO accreditation objectives and criteria^ but 
also to the utility's own standards as defined by policies 
and procedures. The utility conducted self-evaluation 
identifies strengths and weaknesses withii\ each program 
being evaluated. Solutions and action plans are developed 
for zum weaknesses identified and corrective actions taken* 
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PROGRAM EVALUATION 

Program evaluation is an Integral part of all training 
systeiu inodels and is widely used as the principal method for 
determining the effectiveness of w.::::Lning* There are many 
ways to approach prograiri evaluations; there is no single 
evaluation method, that is best for all cases, 

A great deal of reser -^ch has been conducted to identify 
the key components of a training prograin that can contribute 
to its effectiveness. These components arei organization 
and management, personnel/staff^ training process and 
materials^ content of the program, and utilization of 
resources and facilities. The accreditation objectives and 
criteria closely parallel these components. 

The literature Indicates that program evaluation is 
most useful when it is treated as a process---a way of 
decision making^-and when It is applied as such. For 
example, while specific assessment and measurement methods 
can be applied to the evaluation of a given training 
program, the decision-making approach itself is a process 
that can be generalized. Decision areas must be identified, 
sources for data specified, methods of analysis indicated, 
and evaluative decisions made* These steps would make up 
the evaluative process for determining the effectiveness of 
any training program. They too, are common to the 
self-evaluation process . 

COMBINING THE TWO 

Most program evaluations focus on the implementation of 
an individual training program or a cluster of similar 
programs. Rarely does the program evaluation provide a 
comprehensive and in-depth evaluation of the training 
program, the training prooesses, or the training 
organiz^ition. The areas of organization and management are 
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nearly always given the least amount of attention. The 
accreditation self-evaluation process provides an in-depth 
and comprehensive evaluation of the systematic approach to 
training as well as organization and management, training 
staff, and resources and facilities. When viewed 
pragmatically, a properly conducted self -evaluation is a 
comprehensive program evaluation. 

The INPO Accreditation Program is implemented on a four 
year cycles At a minimum a self -evaluation must be 
conducted for a training program every four years. The 
resources needed to conduct an entirely new self-evaluation 
every four years can be significant, and the quantity of 
data and information to be analyzed can be overwhelming* 
Since much of the information collected and used in the 
self-evaluation process is also used for other training 
related evaluations, the eKisting self -evaluation should be 
a living process. This can be done by establishing and 
maintaining a data and Information base for each criterion. 
As changes are made to criteria or as data are generated the 
files are updated. This provides the training staff with 
the most current information to use for conducting 
self -evaluations, preparing reports, or responding to the 
needs of other evaluations. The self -evaluation process 
that has been structured to be an on-going or living process 
can be an effective tool to provide up-to-date data to meet 
the evaluation needs of the training organization. The 
self -evaluation has now been made a part of the management 
process for evaluation and planning. 

THE BENEFITS 

The benefits that a utility may derive by using the 
self -evaluation process as a program evaluation tool can be 
many. While organizational structure, management 
philosophies, and administrative policies vary from 
utility-to-utlllty the benefits to be derived will vary 
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accordingly. However, some of the benefits that can be 
eKpected are as follows i 

o The utility is provided with an on=going thorough 
and comprehensive evaluation of the training 
organization and its processes and programs, 

o The redundancy of evaluations is reduced, 

o A single evaluation program is more cost effective 
than separate evaluation efforts, 

o There is less impact on the time of the plant 

staff — 'less intrusion to collect redundant data, 

o The training staff can prepare evaluation reports 
(including self ^evaluation reports) and respond to 
external agency information needs more efficiently 
and effectively. 

o Training management is provided with up-to-date 
information for planning and decision making. 

Th^ accreditation self-evaluation process is an 
effective evaluation tool. Properly conducted^ the 
self-evaluation is a thorough and comprehensive program 
evaluation. Experience has shown that much of the data and 
information used to conduct a self-evaluation are used in 
other evaluations of the training organisation and its 
processes and programs. The self -evaluation process that 
has been structured to be an on=going or living process can 
be an effective tool for providing up-to-date information to 
meet the changing evaluation needs of the training 
organization. 
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TRAINING EFFECTIVENESS FEEDBACK 



Neal A. Wig gin 
ABSTRACT 

A formal method of getting feedback about the job performance of 
amployees is a necessary part of all our training programs. The formal 
process may prove to be inadequate if it is the only process in use. 
There are many ways and many opportunities to get good faadbaek about 
employee performance* We need to document these methods and specific 
instances to supplement the more formaliaed process • The key is to 
identify them, encourage them, use them, and document the training 
actions that result from them. This paper describes one plant's 
method of getting feedback about performance of technicians in the 
field. 



Ways To Get Feedback 

Recently I heard some training feedback that I wasn't expecting s 
"You trained my group on how to do an inspection ^ and scared them so 
bad that most of them won't do the task at all, because they're afraid 
they will make a mistake I" 

Wow! We thought we had a terrific program, and we do I But we 
created a fear of failure by not being aware of the attitude we were 
creating in the learning activities! We could respond with a remark 
about people refusing to do the tasks they are assigned. But would we 
get any more feedback from that source? Probably not. 

No matter what methods we use to facilitate feedback about train- 
ing effectiveness, they will soon cease to function unless our re-' 
ponses are clear and positive. It would do no good to get defensive 
about our program. If somebody is providing us essential information, 
they need to toow that we are listening and responding. If we are not 
getting feedback now, we must not make the mistake of thinking that the 
clients are satisfied; they may only be convinced that no one is going 
to listen anyway. Silence is our cue to get proactive, iimnediately , 
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There are many ^ood ways to get faedback; the clue is to en- 
courage them, enhance them and use them to make program Improvements. 
To look for faedback only after the training program has been pro- 
vided is to wait five phases too late. Begin with the needs analysis 
phase. What methods do you have for plant personnel to request train-- 
ing? Do you think of that request as an opportunity for feedback. If 
you have progrMs in effect already, the request for training may in-^ 
dicate a gap in the curriculiim. Use the request as an opportunity to 
get the client to talk about training services. 

Who are the trainees? If you already have training programs for 
these trainees, the current request may be already covered by one of 
them. Does the requestor know this? If so, does this request mean 
that the existing program needs to be revised or enlarged? 

Is this request the result of some new tasks? If not, is this 
task on the task lists for this position? Have some priorities 
changed the status of this task? That is, if we are not training on 
this task now. Is It one that we deselected or that had a low 
priority? If so, why is it now on the front burner? 

What prompted this request? An audit? A safety report? An 
employee concern? A plant event? How did we become aware of this 
need for training? If there is some plant or industry event driving 
the request, the caller may be able to provide the necessary reference 
so that you can analyze it for scope and depth of training needed. 

The Formal Method 

I consider anything that tells me that we have a gap In training 
to be feedback about the effectiveness of the training program. Keep-- 
ing that in mind, let's look at the formaliEed method that is expected 
of us, to solicit and respond to feedback about the performance of 
employees after they have completed a training program; it is clear 
that we have to have a formalized method; what we also hr^e to maka 
clear is that such a method cannot be the only method, nojf will it 
even provide the best information out of all the techniques nu use. 
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At Saabrook Statioiij our training procedures call for us to 
follow up training with a request for feedback from job incumbents 
and from supervisors, anytime from three to six months after the 
completion of a training program. Two forms that we use appear at 
the end of this paper. Some of the things we have learned from this 
are the following: 

a. Many of the employess that we trained have not used the 
skills since they completed training* 

b. Employees often comment on training in general ^ whether or 
not they comment about the program they evaluated, 

c. We get a high percentage of response from the survey, A 
follow-up phone call usually will bring in the remainder, 

d* Supervisors return their responses promptly ^ but because they 
have often not seen the trainee perform tasks , the informa- 
tion is not particularly helpful. It is generally true that 
the classes: .-^r^- scheduled on bases that have little to do 
with when tli trainee is expected to perform the work; this 
is partly due to the fact that we are an NTOL plant; we ex- 
pect that when we are planning for outages ^ there will be a 
much greater likelihood that an employee will perform the 
tasks soon after training* 

Feedback in the TSD Process 

We started this scenario with a hypothetical request for train- 
ing, and looked at how we could use that request to get training 
effectiveness feedback* As we proceed through the training system 
development procedure, we have continuing opportunities for feedback, 
not just about the program being developed, but about existing pro- 
grams * 

In the design phase, we create a scope and tentative objectives 
for the new program. Then we involve the client by soliciting cotments 
on the scope and objectives^ i.e.^ who is going to be trained, how 
long it will take to develop the program^ how long we expect the 
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program to run^ what training setting (s) will be used etc. We can 
and should compare this design to other programs which we consider 
successful, with the intent of using what has worked well for us in 
the past« Now we ask the client, "What do you think?" "Will these 
objectives cover the tasks you want people trained to do?" "We think 
this is the optimum number of hours, based on experience with other 
courses of this type; is this feasible for you?" "How many trainees 
can we expect to have at a time, based on current workload?" "Do you 
want to continue with half-day training sessions?" "We could go gull 
days or non-consecutive days if you need some other arrangement?" 
'I^hich scheduling arrangement is best for you?" 

Each phase of the development process should be treated the same 
way, that is, we should be looking for comments that will help not onl 
with the new progrM, but also provide feedback about existing ones, 

Pocumenting the Process 

We have begun a Training Effectiveness Feedback File. In it we 
keep the initiating document and a copy of any response we have made t 
modify training activities as a result of it. Do you have a way to 
get information about employee performance over the course of the 
year? Do you get data from Health Physics about the number and types 
of errors and omissions coimnltted by radiation workers? Is that in- 
formation useful as training effectiveness feedback? Of course it 
is! You use it to change emphasis or to add information to your 
radiation worker training. You know that I You do it all the time. 
Do you document it? Do you have a log or memo file where you can 
track the changes in training that you made as a result of performance 
indicators? If so, put these things in it if they initiated a program 
or a revision to one^ 

1. NoU'-conf ormance reports* 

2. Audit findings on the performance or work or work requests. 

3. Safety reports/lost-time accidents. 

4. Station Incident Reports, 

5. Memo;, from supervisors on any aspect of training, along with 
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6, Memos from you to the file^ about telephone conversations 
concerning any aspect of training. 

7, Minutes of training committee meetings* 

8, Summaries of course critiques from participants, 

9* Formalized feedback forms from employees and supervisors, 
10* Memos from the training department to others , detailing your 

response to feedback. (This is an essential step in insuring 

that others know you really want feedback and will take 

action when it is warranted,) 
11, Responses to Independent Safety Engineering recommendations ^ 

when these address training effectiveness* 
12* Did you say thanks on your responses? If so^ did you mean 

it? 

We have to have a means of getting feedback. 
X^e have to document that we have it* 

We have to use the feedback to modify training activitiesi we 
have to document that. 

We have to insure that the feedback keeps coming. To do that, 
we need to show that we are listening by responding non-def en- 
sively. 

When we do all that, we make the employees and the supervisors 
partners with the training department in quality Improvement* 
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TRAINEE'S EVALUATION OF TRAINING EFFECTIVENESS 



TRAINING GROUP I 

PROGRAMi ^ 

COURSE : ^ 

PHASE- 

LESSON; ^ 

DATE (a) : 

1* What additional training have you received since being last 
assigned to your job? 



2. What unexpected difficulties or problems in job performance have 
you axperieneed? 



3, Has your supervisor given you inetructions different from those 
received in training? YES NO (circle one)* If YES ^ what are 
the differences? 



4, Have there been other differences between the training received 
and what is now expected of you on the Job? YES NO (circle one) 
If YES J please explain i 











5. 


What changes have occurred In your job isnce you 
to it? 


were 


last assignee 










6. 


What tasks do you find easiest in performing your 


job? 




7, 


Which tasks do you find expecially challenging in 


your 


job? 



PLEASE COMPLETE REVERSE SIDE AND RETURN TO THE TRAINING DEPT. 
Fig, 1* Trainee's Evaluation of Training Effectiveness* 
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Looking backj what specific training most benefited you in job 
p er f ormanc e ? 



What kind of errors have been cotmiitted on the job? 



How could training have better prepared you for your job? 



What suggestions would you make to improve the training you re^- 
ceived? 



What additional training do you need for your job? 



Do you have ether comments about training and/or its affect on 
your job performance? 



Date : 

Trainee's Name (Optional) i 

Fig. 2, Trainee's Evaluation of Training Effectiveness, 
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SUPERVISOR'S EVALUATION OF TRAINING EFFECTIVENESS 



TRAINING GROUP: 
PROGRAM 
COURSE I 
PHASE; 
LESSON: 
ATTENDEES i 
DATE(s)= 



1. How well do trainees perform on the Job^ compared to eKperienced 
employees? 



2. What tasks were newly trained employees best prepared to perform? 



3» For which tasks were they inadequately prepared? 
4, What kinds of errors have employees comaitted? 



5, Which tasks require eKcessive tme for trainees to complete? 



6. How do recent trainees compare with those receiving earlier 
training ? 



7, What additional training have trainees received since they were 
assigned their present job responsibilities?^ 



8. Have trainee errors caused equipment damage or failure? 



Has rework by maintenance personnel been required due to per- 
sonnel errors or lack of adequate training? 



PLEASE COtffLETE REVERSE SIDE 
Fig. 4 Supervisor's Evaluation of Training Effectiveness, 
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9. Have employees been commended or warned for unusually good or bad 
j ob perf orinances? 



10. Have you observed unexpected results from training? 



11. Has training created any new problems? 



12, What stiggestions would you make to improve Initial or continuE^ng 
training? 



13. Do you anticipate any changes in job assignments or equipment T^hi^li 
will require additional training or changes in eKlatlng trainijig? 



14. What current training do you consider to be excessive or un- 
necessary? 



SUPERVISOR: DATEj 

DEPARTMENT i 



PLEASE ATTACH A SEPARATE SHEET FOR ANY ADDITIONAL CUmiNTS . REXURW TO 
IHE TRAINING DEPARTOENT. 



Fig* 5. Supervisor's Evaluation of Training Ef f ectivenaaa* 
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F±g. 1. Trainea's Evaluation of Traiuing Effectiveness. 

Fig. 2. Trainea'a Evaluation of Training Ef f ectiveness j page 2, 
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TRAININ G EVALUATION AS AN INTE GRAL CQMpONE^lOr 
TRAINING FOR PERFORMANCS ~ 

ABSTRACT 

A training evaluation system ahould address four jnajor tfSii* 
raaction^ learning, behavior, and results. The training avalujation 
system at GPU Nuelear Corporation addrfeaaea each of thesa thfC^ugh 
practical approaches such as course and program evaluation, ^^he 
major aspects of each practical component of this systim 
desaribed alcng with an organizational atructure to iinpleffi6nt«_ A 
training evaluation system is an important tool of impjoving ^^lant 
performance by assessing trainee performance both in tftining and ii 
the plant. 



INTRODUCTION 

Utilities can be thought of as process cultures which ate 
characterized aa low risk and slow feedback organizationsi f^^r the 
nuclear power industry that changed in 1979 after the 
accident. Nuclear utilities were jettisoned through thatcult^ure 
into high level visability on a variety of issues, as t cisuL^t of 
the accident, the industry now has long lists of auditors Or 
checkers. Today these eKternal auditors attempt to ptoVida piL _ r 
evaluation function for us, in some areas this may Work nllf but 
in areas like training and plant performance, it may not, Tfa ining 
evaluation ia a critical link in the plant performance eqyatior^, 
hence the idea of Training for Performance, internal training 
evaluation must become an integral component of nuclear utilit^^ 
training. Safe, reliable, and economic plant perf orm^n<^e 3re —the 
principle goals for our industry. To be successful, our indus^^^ry 
must become mora serious about generating an internal capabiil^^ty at 
training course or program evaluation. 
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Traditional audits^ whether done by Qk, INPO^ or the NRC tend to 
idantify our miatakes* These processes reflect a belief that 
findinf our problama is aomeona else 'a responsibility. This article 
preaants the conceptual ideas upon which a praetical evaluation 
syatam ia baaed and tha organisational arrangement needed to 
implement* This evaluation system helps us as an industry to 
prevent training caused performance mistakes before they can occur* 

Finally, if performance based training is to be the overall goal for 
training^ than the term "performance" needs to be explained 
further* The use of the word "perf ormanca" does not mean successful 
only performance of trainees in a given training program* This is 
certainly desirable but is not the real reason why the training is 
happening* "per f ormanca* in this case means performance on the job? 
which leads to successful plant performance. The real purpose of 
training is to improve job performance - Training for Performance, 
An evaluation system must reflect that reality* 



The purpose of training evaluation is to provide data to training 
and line management so that Intalligant, prudent, and economic 
decisions can be made about training courses and/or programs,^ 
This process is accomplished by identifying the strengths and 
weaknesses of the training, Mo one evaluation instrument can 
provide all the valid data that we require* 

P*R* Clark, CEO of GPU Nuclear Corporation in his talk in Lyon^ 

France at the Human performance Workahop stated, ■*,* in nuclear 

aetivities, it Is not the admitted lack of perfection but the 

inability or unwillingneaa to learn from experience that ia 
2 

intolerable*" Since evaluation is an important tool of capturing 
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eKperience^ several evaluation schemes or processes exist at GPU 
Nuclear to identify our strengths and weaknesses* The following 
evaluation Schemes are important tools in ensuring that we are 
training for performancei 

^ Program 

^ Course 

^ instructor 

^ Trainee (while in training) 

^ Trainee (once back on the job) 

Program Evaluation 

The two most common labels used to describe training are training 
programs and training courses. These two labels describe training 
that differs primarily due to length. Training programs are usually 
large scale training efforts which we use to develop new personnel 
by certification or qualification to do specific jobs* Since each 
varies significantly in length, it is more practical to approach 
their evaluation with appropriately scaled processes. This ensures 
that our evaluation activities will remain manageable and 
economical* some typical training programs include the Non-Licensed 
Operator and the Radiological controls Technician training programs* 

TO better understand the compleKity of a program, it may be helpful 
to briefly describe its anatomy* Many training programs are broken 
up into segments or phases such as clr :room, laboratory/simulator, 
and OJT* The classroom may be furthe ubdivided into fundamentals 
or systems, it is not unusual for the to be further broken into 
specific courses such as heat transfer, pump principles, rigging, or 
the RHR system. 
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instructor involvement is another aspect about training proframs 
which generally differs from individual courses* The conduct of our 
largest training program generally requires the coordination of 
several different instructors* This aspect can often complicate the 
evaluation but is an important facet that should be recognized and 
addressed where possible. 

The continuity of a program is generally difficult to achieve when 
many instructors are assigned to teach, program evaluation should 
be sesii^iUive to this issue, and point out problems in this area when 
observed. 

Since programs are generally made up of courses which are frequently 
grouped into segments or phases, it is important to evaluate how 
well the phases are tied together, program evaluation should not 
only collect data about specific pieces of the training but also how 
well these are integrated together, i have found that in many 
training programs the phases are generally well designed, when 
viewed by themselves. However, frequent problems are found in how 
the phases link together or the instructors utilize the content 
taught in the earlier phase. 

The logistics of evaluating such large blocks of training requires 
the evaluator to carefully determine the proper quantity and level 
of data to collect. If too detailed^ then the evaluation process 
becomes unwieldly and may demand an unreasonable amount of 
resources to complete. Most of us are familiar with the IMPO Self 
Evaluation Reports ^(SER) and the resource requirements we needed 
to complete them. The SER is a form of program evaluation, 

I have several eoncerns with the INPO SER process if it is used as a 
utility's program evaluation process as i have heard suggested. 
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First it's not vary suitable for those programs that are not 
presently under the accreditation process such as General imployee 
Training program* Second^ implementation of the SER required an 
extremely large and costly resource commitment^ particularly in 
manpower. And thirds the SER process is an open process as opposed 
to forced choice. The criteria are shaped in open--ended terms which 
made it difficult for less than fully experienced evaluators to 
arrive at consistent conclusions, i base that conclusion on the 
last three years of industry experience. Many utilities had to 
resubmit their SER's to INPO before they were acceptable so a site 
visit could be scheduled. There are a variety of reasons that could 
be attributed to resubmissions, but I believe this format to be one 
of the more important reasons, 

_ _ - _ . 4 

NUREG 1220, Training Review Criteria and procedures, has recently 

been released as the NRC's check on the INPO process. This process 

is also another example of a program evaluation instrument, 

NUREG 1220 appears to overcome my last concern about the 

accreditation process and SER format, NUREG 1220 evaluates each 

objective on a numerical scale of 1, 2, 3* These numerical criteria 

are defined in the context of the objective (see Figure 1), 

The NEC expects that two evaluators can complete a 1220 evaluation 
in two to four days. If a program lasts up to a year^ that means 
that the evaluators are seeing an extremely small part of the actual 
course. They would essentially be doing an evaluation of the 
training documents. 




Figure 1^ ixample from NUREG 1220 
Review Guidance 

2.2 Are learning objectives derived from or related to the 

knowledge, skills, and abilities needed for succassful job 
performance? 

Scoring Guidance 

For the three to six tasks selected for the review, score this 
question as a: 

1* if learning objectives are not derived or related to the 

skills, knowledge, and abilities that enable the trainee to 
perform these tasks. 

2* if learning objectives are derived from or related to the 

skills, knowledge, and abilities that enable the trainee to 
perform tasksi 

^ Selected for initial training, 

^ Related to emergency/abnormal operations 

(the only exceptions being any skills/knowledge that are 

assumed to 
be entry-level skills/knowledge), 

if learning objectives are derived from or related to the 
skills, knowledge, and abilities that enable the trainee to 
perform all the tasks selected for this review (the only 
exception being any skills/knowledge that are assumed to be 
entry-level skills/knowledge). 
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1 am leary of paper drill exercises. The reality of the actual 
training im normally quite different from that cryptically described 
in the training documents* i can't overstate the fact that Training 
for Performance is our goal. To achieve maximum validlry, 
evaluations should be done in the arena where perform?-^'^ ^-^curs: 
in the classroom/ in the simulator or lab, and after uz^- , .cation 
on the job. The program evaluation instrument that i? a part of our 
ayatam accomplishes this since it directs the evaluat^r to follow 
the entire teaching cycle of the program. The evaluator monitors 
the effectiveness of the training through a reasonable number of 
classes through all phases. This includes going on shift to observe 
trainees during QJT* Additionally, it can also be done by one 
person I 

The program evaluation instrument^ which I have designed evaluates 
theae long qualification oriented training programs, it examines 
each program in eight areas (see Figure 2). After initial 
experience/ the instrument was revised and now includes some 
objectives/criteria of the accreditation process. One of the 
advantages of our evaluation Inatrument format is the use of the 
Likert scale {see Figure 3), Many questions are framed in a forced 
selection format. These are generally followed by a rating question 
which rates the overall quality as seen by the evaluator. An 
opportunity is generally provided so the evaluator can follow up 
with narrative comments if desired. 



429 



Figure 2i 



program Evaluation Instrument Major Categories 
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III* 


program outline 
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Student Evaluation 


VII, 


Training staff 


VIII* 


Miscellaneous 



Figure 3: Typical Page from Program Evaluation Instrument 
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This instrumant ia formatted to simplify the creation of the 
evaluation report* The report ia written as the evaluation 
instrument ia being completed. Experience has found thia aspect a 
real plua in reducing administrative resourcea needed to conduct a 
major program evaluation. Additionally ^ it standardizes the 
evaluation reports format from one evaluation to another. 

At GPU Nuclear y the major focus of our efforts over the last two 
years has been oriented towards obtaining accreditation for our last 
15 major training programa* The major part of thia work was in the 
analysis and design phases of TSD which left little time for program 
evaluation,^ Since we see the value of program evaluation, we are 
now placing considerable emphasis on more formal evaluations. 

Course Evaluation 

Many of our large training programs run six months or more on 
average. These programs are made up of many small courses as 
already described. What doea one do when problema emerge in only 
one or two couraes within a long training program or in a very ahort 
program like General Employee Training? 

AS previously mentioned, the program evaluation instrument is big 
picture oriented and wasn't designed to collect detailed data. To 
overcome this limitation, a courae evaluation process was 
designed,^ Course evaluation should achieve aeveral objectiveai 
1) collect detailed-specific data, 2) obtain maximum input from 
involved participants, and 3) collate data into a narrative aummary 
for quick diapoaition* 
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For course evaluation, whether for eKisting-mature courses or for 
new ones just being piloted^ it's important to get as many 
independent points of view as possible. The trainees themselves 
have a unique vantage perspective. But the same can be said of the 
instructor (s ) . Many excellent Instructors have cQmplained that 
evaluation activities typically Ignores their insights. Soma even 
feal "ganged up on" so to speak ^ since we work in a highly regulated 
industry. The course evaluation process being described here 
mandates that the instructors have a formal input* 

The major players in course evaluation arei the trainees, the 
instructor (s ) , and a tbltd party individual called the course 
evaluator. To standardize the process from application to 
application y a series of forms exist to keep the focus of each 
activity on target. The course evaluator and one or more of the 
instructors review the major tests after administration, it is 
suggested that this review be done independently. Finally, the 
course evaluator pulls everything together into a narrative summary 
report. 

This summary highlights the major strengths that were observed. it 
also identifies the most significant areas of concern and who made 
the identification. This last aspect is important to add to the 
validity and/or comprehensiveness of the finding. The lead 
instructor or supervisor has an opportunity to review the narrative 
summary before a face to face meeting is held. This step is very 
conducive to generating trust and rapport, it does not, as some 
might think, lead to compromise, in fact, that is the process INPO 
utilizes in finalizing its accreditation report after its site 
visit, in our process, if no serious disagreements exist, then a 
meeting is scheduled. 
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This meeting should be attended by the evaluator, instructor ( s ) , the 
supervisor, and senior training manager. At this time, the issues 
are discussed. First the strengths of the course are reviewed then 
the concerns. Not all sited concerns will result in follQw--up 
actiona. Much depends on the severity of the concern and how 
frequently it was cited. Mutual agreement is sought to strengthen 
the commitment to the follow-up effort, 

pollow^^up actions are planned for those concerns that are accepted 
in this review meeting* These are classified into two categories^ 
those needing correction before the course is retaught and those 
that can be addressed long term. Accountabilities are assigned to 
individuals to accomplish these actions (the individuals who have 
the capability to complete these revisions are generally the ones 
attending this meeting). The process is problem solving focused. 
Both the evaluator and the lead instructor or supervisor make a 
joint presentation to the senior training manager. This provides 
them with an opportunity to show off the quality of the course by 
providing an objeGtlve picture of the course's current status, 

TO date, nearly twelve course evaluations have been completed, 
several evaluated newly developed courses taught for the first 
time. The attitude towards course evaluations by many of our 
instructors and supervisors is changing as a result of the methods 
I've described partieularly getting them Involved, That is 
something i didn't expect but am very grateful for, 

in our course evaluation experience, several interesting findings 
have been identified, one important finding deals with theory, in 
our industry^ it appears that in many cases, we often provide too 
tnuch theory for the practical aspects being taught, in several 
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evaluations we've completed, the trainees cited this repeatedly. 
And this was often independently noted by the course evaluator. It 
became difficult for learners to see the forest for the trees, in 
another area, findings pointed out insufficient opportunity to 
practiee practical skills or tasks. Sequencing the material was 
taught in was another rather cominon finding identified* More 
practical suggestions were found via trainee and course evaluator 
comments or reactions on how to correct sequence problems. After 
all, our instructors are subject matter experts, and it's difficult 
for them to sense the impact one sequence can have over another on 
learning effectiveness* But trainees and course evaluator s can, in 
retrospect, most instructors agreed to be more sensitive to 
course-content sequence* 



This is an important area in the evaluation process* Most utility 
training organizations expend more energy in this area than any 
other, our own work at GPU Nuclear is strongly oriented towards 
evaluating group paced training, I hope that we will be able to 
shift towards diagnostic assessment as the individual enters 
training* That way he or she will only receive training in areas 
whan a true need exists* This does not have to be done by expensive 
computer based training. It can be done economically with more 
traditional approaches. 



This area has been an especially fruitful area of pursuit for us. 
With the finalization of our TSD process, we saw this as an 
important evaluation activity. in the past year, the Educational 



Trainee ivaluation (while in training) 



Trainee Evaluation (once back on the job) 




Devalopinent section (EDS) of the Training and Education Depactment 
has begun playinf a more aetive role in this kind of evaluation, 

in 1986 the Oyster creek operator Traininf Manager requested that 
EDS do the poat training evaluations (as they are sometimes called) 
for one of his recently completed licensed operator training 
programs. The tralneea had already been on shift for about six 
months after qualification* The primary reason for our assistance 
was to improve the objectivity of the collected data. The first 
attempt by Operator Training at getting the data was made by mailing 
a questionnaire to the participants and their supervisors. Returns 
were difficult to get^ and the information collected was of little 
value. The former trainees and their supervisors appeared to be 
uncomfortable with filling out perhaps ^another form." The operator 
Training Manager thought In person interviews would likely be more 
useful. He was uncomfortable with having his instructors ask former 
clients^ so to speak^ how the program worked in a face-to-face 
interaction, in essence, the instructor would likely feel he was 
asking the trainees to now rate him personally. 

EDS conducted the interviews anonymously using a series of questions 
in the evaluation procedure. The operators were a little nervous at 
firsts but the Educational Development Coordinator or myself were 
careful to stress the purpose of the evaluation. The purpose wasn't 
to evaluate the former trainee since they had been so stringently 
evaluated during their training, to the contrary, it was their turn 
to help evaluate their learning experience, in addition^ to 
minimize the threat, they were told they were not required to ajiswer 
questions they were uncomfortable with, we were pleasantly 
surprised that no one refused to answer any of our questions. We 
used the interview questions from the procedure. 
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Thay willingly anawered each question even one which asked them if 
they had made mistakes back on the job. To set the record straight^ 
we are proud of all our training programs and our instructors. 
However, we learned a great deal about the program and identified 
about six key areas where it could be enhanced. 

In one area, several ex-trainees told us they had a great deal of 
difficulty absorbing or retaining all the system knowledge which was 
taught in one continuous block. We asked them to provide 
suggestions. The reply we got was to integrate OJT into the systems 
phase. This recommendation was readily acseptad by the Oyster creek 
Operator Training Section, 

instructor Evaluation 

One other important aspect of our evaluation efforts is instructor 
9 

evaluation. This evaluation area has been in place for many 
years now and is working quite well. Each instructor experiences 
several evaluations each year. Most of these are by his or her 
supervision to include section manager and manager of plant 
training. These primarily focus on technical content evaluations 
with secondary emphasis on the quality of the classroom 
Instructional skills, 

EDS performs one annual evaluation of each instructor. The primary 
focus of this evaluation is on the instructional skills. Lately the 
focus is shifting from what was almost a singular emphasis on 
teaching skills towards a balance between teaching skills and 
curriculum development. This latter area is exhibited through 
behavioral learning objectives and teaching materials such as lesson 
plan and trainee texts. Annually ids evaluates all completed 
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instructor evaluations to identify the overall strengths and 
weaknesses of our instructors* Two of these analyses have been with 
good results. The identified areas of difficulty are relatively 
minor, in our latest analysis, we have seen generic weaknesses in 
the use of audio-visual media and in application of effective 
questioning techniques, two advanced instructor training modules 
are under development as a result of identifying these needs through 
evaluation * 

ORGANIZATION 

If your plant has been through the accreditation process^ you have 
developed and documented your systematic approach to training* It's 
effectiveness is highly dependent on your organigation and how 
accountabilities are balanced. At GPU Nuclear, we have evolved to a 
team approach to training program developments subject matter 
expert^ instructor, and instructional technologists. 

Special attention is also being placed on knowledgeable but 
reasonably objective evaluations being done to support the 
department efforts, 

TO accomplish this, the Director of Training and Education 
Department (TED) established EDS, As our development efforts shift 
from TSD's front-end analysis phase to evaluation^ EDS is continuing 
to play an active role. Figure 4 shows the TED'S organization. I 
have added the dotted lines from EDS to the other managers to show 
how I perceive the emerging operational philosophy. EDS is a small, 
but dynamic^ department resource with three (3) Educational 
Development Coordinators, one (1) Audio Visual Specialist (who 
operates our video tape studio), a Secretary, and the Manager, 
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pifure 4i organization of GPU Nuclear »s t&e Department 
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The manager/coordinatora are training-learning spaeialists who 
posses advanced degrees. Three were former educators with both 
secondary and some college teaching experience, in addition/ three 
have military training experienGee to balance the educational 
perspective with practical utility experience^ one coordinator has 
over 30 years with the utility* He was one of the original oyster 
Creek startup staff whose nuclear experience predates 1967. 

Over the past two years, i have made a serious effort to further 
develop and enhance their training development and evaluation 
competencies » i achieved this by coaching sessions and OJT which 
closely parallels apprenticeship* 
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This small cadre of training specialists supports approKimately 80 
inatructional staff located at three training sites and supports 
training development and evaluation efforts for corporate clients* 
Since our resources are quite small, we have saught ways to increase 
the span of our reach* a module was recently developed on course 
evaluation which was taught to departmental instructors* we have 
used instructors/supervisors as course evaluators. The educational 
development coordinator coaches them through the process as it is 
done after taking the module mentioned above. 

THEORY 

Evaluation has always been the soft underbelly of the educational as 
well as the industrial training function. There are no shortage of 
opinions and concepts on what it is* But training staff generally 
dread being assigned to do evaluations, i believe this is 
principally due to a lack of tangible-objective mechanisms or 
processes * 

in the early 1960 's, Kirkpatrick, ^° one of the early thinkers on 
industrial training evaluation, developed an evaluation model. 
Almost all other models available today are based on his original 
plan. The Kirkpatrlck evaluation model has four major components^ 
1) reaction, 2) learning, 3) behavior^ and 4) results. 

Step 1* Reaction 

Reaction is based on the premise that adults are knowledgeable and 
experienced* They generally have Important things to say* Whenever 
possible, an evaluation system should tap into this valuable 
resource, in our course evaluation process, we have reaction forms 
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for the trainees, the instructor (s) , and the course evaluator (This 
process of soliciting reactions Garries over to interviewing the 
former trainees and their supervisors four to six months after 
qualification in our model). Reaction evaluations are relatively 
easy to do which is likely why it is so popular* \^en reaction type 
evaluation constitutes the bulk of the evaluation system^ there are 
tendencies to draw conclusions which are based on assumptions. To 
prevent arriving at questionable conclusions, evaluation must be 
based on other criteria as well such as learning* It isn't unheard 
of that training courses or programs have been cancelled or 
underwent drastic revision based on one negative comment from a 
senior management person alone. Prom the other extreme^ there have 
been training courses which have received strong positive reactions 
which only entertained. These courses would have been dramatically 
changed if other data had been available. Neither of these extremes 
are desirable which motivates us to evaluate on other levels as well 



Learning is not guaranteed just on the basis that a program has 
received a favorable reaction. Learning in this context implies 
retention of the facts, principles, and skills that were taught and 
understood by the trainees. There are five (5) guidelines that 
Kirkpatrick recommends be observed for measuring the amount of 
learning. 



Learning should be measured quantitatively. 
Where possible, pre and post tests should be used. 
Assessment must be based on behavioral learning objectives 
Where possible, a control group should be evaluated and 
compared to the group that had received the training. 
Where possible, the results should be analysed with 
statistical procedures for correlation and confidence 
verification for learning. 



Step 2% 



Learning 




The principal methods which iifi^lainent this guidanca ate generaX iy 
used in the nucl&ar power Indus ^try: elassroom demonstration ar^ a 
performance as weal as pencia a mnd paper tests. 



Step ^3 I Behavior 



Knowing a paftiduiar prinaiple © or possessing a skill does liot 
guarantee that thmsB will be \ib& m6 out on the job. over thelasfc 
three yeari, numerous artiGlea miave been written in traitiliig 
journals about the problem off l&earning transfer to the woikplaGe 
environment. PeChaps that is Oraie of the biggest challent€S thafc 
remains to be sol\red for many nuclear utility training ptfigtams^ 
Many millioni of dollars have P^en spent designing the £imt 
training programa with excellente" training materials^ faollities^ and 
instructors only to find that Cr^ained or gualified peopla do not 
perform back on the job (in SQine^ cases ^ they are able to but 
prevented from per forming for ^t -her environmental constrainta) . 

Kirkpatrick'a evaluation system: provides five (5) guidelinii tha^ 
should be Goniidered when evalua^^ting behavior. They ares 

A systematiG appraieal should be made of on-the-job 
perfocmanca on a beffor& e and after basis. 

The appr^iial of parf ©ctxmance should be made by oiii or more 
of the following groups^ (the more the better )i 

' Tha person receiving the training 

- The person's super vl^wor 
The parson's subordit^^ates 

- Tha person's peers oar other people thoroughly familiar 
with hLs or her pe^^Eormance 

^ A statist leal analysis should be made to Gompare 

perforiaaca before and after to relate changeg to the 
training pcogram,* 
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® The poat-ttaining appr^aisal should be made three months or 
more after the tralnl^ig so that trainees have an 
opportunity to put int=o practice what they have learned* 
SUbsequant appraisals may add to the validity of the study, 

* A control group (not eceiving the training) should be 
uaed.* 

The standard itema are not now «^common to our industry. They are 
areas which should riciive futur —a consideration if additional 
improvements are going to be niadHe* 



Step 4 ^ : Results 



What is the purpose of the tr^m^ing course or program? Is it to 
obtain licenses? Produce a apea^^ific number of qualified Individuals 
for a particular job poiltion? ? Th^se have been some of the 
traditional measurea of the resU/.lts for some of our nuclear industry 
training programs, lean pet aona. ally remember several years ago 
working with a very practical an^ d hard nosed training manager. Our 
discussion centered on identifyls ng the licensed operator training 
program resulta. The discussion covered considerable territory. 
Shortly after that? he raceived ^the pass/fail results from his 
plant^s last ^^^^ of NRC sxams, a majority of his operators had 

failed theit ..^ms- This same Bo<-enario repeated itself with the 
ne^t group of candidates. That tSiard nosed and practical manager was 
very much beside himsilf. He kn^^w hir. job was on the line, if the 
training department couldn't obt^Min licenses, then there wasn't much 
purpose for it as far as senior ^»lant management were concerned, no 
one knew this fact of life battfic^ than that training manager. 



However, to succeed in the face c=if our new industry challenges, we 
must establish better goals and ^^bjectives for our training courses 
and programs than just the numbfir^ of licenses we obtain, we must 
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look for bettec measures that reach out into the real environment of 
the plant. For mechanics^ the number of rework tickets and the time 
it takes to complete routine jobs are examples that may be 
considered* The utility Nuclear Power oversight Committee 
(UNPOC)^^ has just been organised. one of its prime objectives is 
to improve the operational performance of nuclear power facilities, 
one of their primary goals is stated as a question. "How can the 
nuclear utilities accelerate the achievement of exemplary 
operational performance by all U.S# nuclear electric generating 
units?** 

initially ten (10) areas have been identified by UNPOC as prime 
candidates for plant performance. Training organisations should 
start with these and expand them while looking for other more 
applicable results oriented Inraicators for their training 
evaluations. in the meantime, training organisations should 
aggressively evaluate in terms of reaction, learning, and behavior. 
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USING EVALUATION AND FEEDBACK TO IMPROVE PERFORMANCE 



Steven Ketcham 



ABSTRACT 



Training programs throughout the nuclear industry 
have undergone considerable ehanges in the last several 
years* TMI caused us to go back and take a hard look 
at existing training and make modifications where 
necessary to conform to new regulatory requirements. 
INPOj through Che perf ormance^based training concepts 
has had us again go back and look at our training 
to ensure that each identified performance task is 
addressed. Because of these and other related re- 
quirements 5 I believe that all of our training programs 
have improved. However ^ to ensure that these programs 
continue to provide relevant and meaningful trainings 
evaluation and feedback must be an integral part of 
the process^ 
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EVALUATION AND FEEDBACK 

Evaluation and feedback are essential ingredients to a successful 
training program- They provide both a direct measurement of the 
effectiveness of the training program and also solicit criticism 
and comments from the students* 

Ivaluations are given to test a person's knowledge of a specific 
topic or his ability to perform a given task. There are several 
different types of performance evaluations utilized throughout our 
training programs: WRITTEN, ORAL, and PRACTICAL (shop, lab, simulator, 
on-the-job qualification) = Each of these has a specific purpose 
and place in the evaluation process. 

Written examinations provide for short^ and long-term retention 
evaluation, but may not allow the individual to fully express his 
knowledge or mastery of a subject* 

This is where the oral evaluation or examination takes over. 
It allows the individual to express, in his own words 5 his under- 
standing of a topic* This gives the evaluator a better view of his 
overall knowledge and allows the evaluator a chance to thoroughly 
investigate the individual's knowledge through additional questioning* 
Tliis process also provides for immediate feedback to the trainee* 

Pri^tical evaluations also have a very specific place in the 
evalua. n process , especially where performance related tasks such 
as maintenance 5 calibration, and equipment operation are involved* 
Practical evaluations are given on the specific equipment and tools 
utilized by the trainee in the shops 5 laboratories, and simulators 
or on the job utilising the actual plant equlpinent- 
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Written examinations are a very important part of the evaluation 
process. Since a majority of the training that we conduct is through 
classroom presentations and requires the student to obtain a specific 
level of knowledge of the presented material, the written examination 
^ffo^rds us a convenient mechanism for measuring how effective our 
training has been. However, in order for the examination itself 
to be effective, it must be based on the instructional objectives 
derived from the job analysis and task listings. 

To ensure that the written exam is providing the desired evalu^ 
ative information, a multiphase process is employed. First, each 
test question is referenced to a specific instructional objective. 
Second, the exam i^s reviewed by another member of the training de- 
partment responsible for that training. This helps to remove any 
bias that may have been present during the development of the exam. 
The examination is then approved by the Training Supervisor of that 
department. This approval may also take the place of the second 
member review. Certain exams, such as the license operator requalifi^ 
cation exams, are also reviewed by an instructional technologist. 
This review looks at format to ensure that each question is in- 
s tructionally accurate. In addition, each requalif ication exam is 
reviewed and approved by the station General Manager. This development, 
review and approval process ensures that each examination will accurately 
measure the appropriate knowledge and/or skills of the student. 

Oral evaluations or examinations are another method of measuring 
the effectiveness of training. Through oral examination, the in^ 
structor is better able to determine the actual entent of the student's 
knowledge . 

Many people have difficulty in taking written exams, especially 
when they are required to write out their throughts or a sequence 
of operations. Oral exams are a great supplement to the written 
axam and provide for immediate feedback to the student when a wrong 
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response Is obtained. However, like the written exams the oral exam 
must also be carefully prepared and reviewed to ensure that it accu^ 
rately evaluates the appropriate training material. This includes 
preparing the questions in advance and ensuring that each question 
is referenced to an instructional objective. Assigning a point value 
to the questions and indicating the key items expected in the answer 
will help in maintaining objectivity and provide for immediate feedback 
to the student when the correct answer is not obtained* When a re^ 
sponse requires the use of a specific piece of equipment or other 
reference material 5 ensure that it Is made available to the student 
or that he knows that he is responsible to obtain that material in 
order to answer the question. Just being able to obtain the material 
may be a key factor in being able to answer the question^ and would 
be worth a majority of the point value. For this type of situation, 
this should be indicated on the evaluation form. As with all evalu^ 
ationsj the oral evaluation form should be well documented and retained 
on file for future reference (Figure 1). 

The third type of evaluation used to measure the ef f ectlveneF.s 
of training is the performance evaluation. This is similar to the 
oral examination but generally involves the student demonstrating 
proficiency of a specified task using appropriate procedures^ tools 5 
equipment or simulators as necessary. Performance proficiency is 
an important aspect of training in the areas of Maintenance, Chemistry, 
Radiological Protection and I&C, as well as Operations. We are seeing 
a much greater emphasis being placed in these noWj then we have In 
the past. This can be attributed to the fact that we have had to 
take a much closer look at these areas and evaluate the needs of 
the individual to complete the INPO requirements of the job task 
analysis. The need to evaulace each student in his performance of 
each identified performance task, has caused the proficiency level 
of the individuals to be raised greatly by the time he is assigned 
to the plant as a technician. Although he may still be required 
to complete some in plant qualifications, he is a more productive 
member of the department and requires less direct supervision. 
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The performance evalustionj as trie oral evaluate ion^ must be 
well documented to include the appropr late procedure s , necessary 
equipmenc and the minimum requirements to successful ly complete the 
performance evaluation (Figure 2), 

For any of these forms of evaluat ion to be effe ctive In the 
training procesSj they must be used as an eva luat lrj£ tool. This 
means that not only will thay measure the student *s knowledge of 
a specific topic or his ability to per form a specific task, but the 
evaluation must be reviewed to decermixie if the resu ^ts are in line 
with what is expected, In a word, if what we get is not what we 
wanteds then we must deterniine why. Xtiis is cominQni^^ referred to 
as feedbacks 



Feedback is the second ingredient to a success 1 training 
program. It comes in many forms. Froxti an informal conversation 
with a technician performing an instrument calibrati^^n who states 
that using a specific piece of test equxpment during training would 
have helped him in the fields to the fornial End of Course Feedback 
Form obtained at the end of every class. No matter ^^hat type of 
feedback used, it ia essential that it is documented^. reviewed for 
relevance and incorporated into the training prograin^ 

Several types of feedback may be used throughout a training 
program at any one tlmi. We use three %7arious types of formal 
feedback in our programSi 

First, there are the Inrunediate^lnd of Course Fee <iback Forms 
(Figure 3). These forms are handed out by the Instru-ctor and filled 
in by the students at the end of each program. For 1 ^ng duration 
programs, several feedback forms may be requasted throughout the 
program to get a repreaentat ive sample of the student 's comments. 
These forms are divided Into three major categories ^ 
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o program content and training material 

o instructor competence and technical knowledge 

o evaluation criteria 

In addition, there is room for the student's written comments. 
These feedback forms are reviewed and summarised by the Training 
Supervisor. He will generate specific action items as necessary 
to correct any deficiencies noted. This summary and action plan 
are then reviewed and approved by the Principal Training Supervisor 
and Department Head. 

It is then placed into a tracking system where it is followed 
until completion of all action items (Figure 4). Upon compietlon, 
the summary and action plan are filed as an integral part of the 
training program (Figure 5). 

Second, there is the Post Training Feedback Form (Figure eajb). 
These forms are sent out 90 to 120 days following the completion 
of a training program. One form is sent to the student and a second 
form is sent to his or her supervisor. The student *s form asks him 
what he expected to learn from the training and whether or not the 
program met his expectations. The supervisor's form asks what he 
expected the student to learn from the training and whether or not 
the student's performance has Improved as a result of the training* 
The feedback forms are returned to the Training Center where they 
are reviewed and summarized by the responsible Training Supervisor. 
The Training Supervisor will again produce an action plan to ensure 
that any significant comments or deficiencies are incorporated into 
the program for future presentations. The feedback sunmary and the 
action plan are reviewed and approved by the Principal Training 
Supervisor and the Department Head and they, too, are tracked until 
completion (Figure 4, Figure 7). 
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A third source of training feedback is the Performance Ivaluation 
of Training Effectiveness or the P*E.T*E. Program (Figure 8j Figure 9) 
This program provides for a measured evaluation of specific tasks 
by supervisory and management personnels as well as training personnel 
Selected tasks from the task listing are scheduled to coincide with 
plant activities. This allows the evaluation to be performed on 
a specific task being performed by a qualified individual under actual 
plant conditions using the approved plant procedures* These evalu= 
ations are reviewed by both the training departments as well as plant 
management personnel. As with all feedback Information , these are 
summarised and appropriate action Items generated. 

The P.E.T.E* Program was established in response to an INPO 
station evaluation item. This program has been in place since October 
1985* In the beginning, this program required only an annual evalu^ 
ation of each job classification groupj machinist , chemists etc. In 
April of 1986 1 this program was integrated with the Salem station's 
on^the=Job evaluation process. This now causes j on the average ^ 
t^.o observations a month for each job classification group. As of 
September 1986 ^ twenty observations have been reported j of which 
three have resulted In modifications to the training process* 

In essence i the station publishes a schedule of on^the^job 
evaluations to be conducted during a given time period. Using this 
schedules training department personnel will schedule a P.E.T.E. 
observation* To ensure that a single task is not being observed 
each times a log is maintained. Using this logj the Individual will 
select a specific task to be observed. Upon selection of the taskj 
the Training Supervisor will approve the selection. The instructor 
shall then review the appropriate lesson material to determine the 
"as taught" method used for training of the task. During the obser- 
vatioui the evaluator does not Interfere with the performance of 
the task unless, in his opinion, there Is a significant and Immediate 
concern for the safety of personnel or equipment. In this casej 
he may take whatever action Is necessary to prevent personnel injury 
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or equipment damage. As with all feedback forms, the completed evalu- 
ation is forwarded to the Training Supervisor for review (Figure 10). 
Corrective actions are noted and forwarded along with the evaluation 
to be placed into the tracking system. Completed evaluations are 
filed for future reference. 

Using a combination of each of these feedback programs, a large 
sampling of comments and criticism, as well as a visual indication 
of the training program's effectiveness is possible. Incorporation 
of the action plans into a tracking system ensures that necessary 
corrections or modifications to the program are made. 

Another method of using evaluation and feedback to improve 
performance is through the use of Training Review Groups. Training 
Review Groups have been established for each specific area of training 
{Operations^ Chemistry Radiation Protectiong Maintenance 5 I&C, and 
Engineering). These groups are comprised of the training Department 
Head and the Principal Training Supervisor, the plant Department 
Head and supervisory personnel as necessary , and a representative 
of the bargaining unit where applicable. These groups meet at least 
quarterly to discuss upcoming schedules 3 class loading, course feedback 
comments and course content. In additiona these groups review changes 
to the task listings to ensure accuracy. The use of the Training 
Review Groups has caused the plant personnel to become more involved 
in training so as to give them a feeling of ownership. 

Alone, each of these areas would have a small impact u ' ■ 
overall training process. However, through a sustained, cr-^v I - 
efforts the evaluation and feedback programs help to ens >h ^ 
we are providing effective and relevant training to each iu^ii. - 
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ORAL EXAMINATION PRACTICAL DEMONSTRATION 
GRADING FORM AND INSTRnCTIONS 

1, On the Form No, 301-1/ fill int 

a* Student name 

Evaluator name(s) 
c* Course 

d. Date of examiriat ion 

e. After the examination^ number the pages (1 of 2, 
2 of 2, etc, ) 

f* Location of examination 

2, The emphasis on the oral examination should be on task 
performance. Each task discuseed or simulated shall be 
listed in the "TASK/QUESTION" column* The trainee shall be 
evaluated on his/her ability to perforin (simulate) specific 
tasks in accordance with the applicable procedures, 

3, Evaluated questions/tasks must be graded Sat* (S) or Unsat* 
(U) by marking an X in the appropriate block. All Unsat. 
(U) grades require comments. 

4, If on the spot remedial training is conducted and a question 
is repeated, regrade it by marking a / in the appropriate 
block. 

5, On the final page/ mark the appropriate box for the overall 
grade/ sign as the evaluator and comment as necessary- 

€• Conduct an examination review and have the student sign the 
final page- Sign as the evaluator. 



?• File in accordance with procedures. 

EKLC 



Attachment 1 DATEs 09/11/85 



V,B,5J0 



Fi g L: rc 1 



ORAL EXAMINATION FORM 



idant 
irse 



Page 



of 



Location 



Task/Question Summary 


S 


□ 


Commants 


















































































leral CornmantSi Overall Eval, 









ATTACHMENT I 
Page 2 of 2 



Evalua tor Signatutra 



Form No. 301-1 



DATSi lQ/17 /84_ 
REV, I 4 
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POINTS 


^ni— 157 fl2^LAB 040^05 
SULFURIC ACID i¥ TITRIMETRY 


MOO-05-05000 


POSSIBLE/ 
ACHIEVED 


NOTE I USE MANUAL TITRATING BURET 


AFETY OF 
ERSONNEL 


15 / 


EYE PROTECTION (2 PTS.) 
RUBBER GLOVES (1 PT,) 
LAB COAT ( 2 PTS * ) 
MSDS INFORMATION (10 PTS,) 


AFETY OF 
QUIPMENT 


5 / 


CAREFUL HANDLING OF BURET (5 PTS,) 


PERATION OF 
QUIPMENT 


10 / 


PROPER FILLING TECHHIQUE (5 PTS.) 
PROPER TITRATING TECHHIQUE (5 PTS.) 


ET UP OF 
XPERIMENT 


10 / 


STANDARDIZE NaOH TITRANT CORRECTLY (5 PTS.) 
PREPARATION OF KAP (5 PTS.) 


OLWH THE 

ROGEDURES 


15 / 


PROPERLY MEASURED SAMPLE (S PTS,) 
TITRATE TO PROPER END POINT (S PTS.) 
DATA RECORDED CORRECTLY (5 PTS.) 


ATH 


0 / 


INCLUDED IN CORRECT RESULTS 


ORRECT 
ESULTS 


45 / 


CONCENT^TiON OF UNKNOOTIS - % RECOVERY i 
90% - 110% 

CORRECT NORMALITY (10 PTS, EA.) 
CORRECT PERCENT (5 PTS, EA, ) 


INAL 
RADE 


100 / 
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Figure 3 



COURil TfTLE _ . LOCATfON 

MAtm OF iNS^ucTO _ . 

OATECS) SF ArfENDAIiCI ^OUR NAME {Q^TiONAL) 

QiREeTlONS i 

PIUM elrfl* ysyr rcppsni* on iQSim of f - I f&r iaeR guiaTien, if /sy f«j that ysu Ci; ant rssesna 
ts a p^rtleulir guvstlsfl, IM¥« It unftniwvrvd. ft fh«r* eft stHBf arsai that ysu wish ta ^^Sftsi, 

ptHM <^ fs sn a saparatt sh»#T and aftaefi tnii F»«abaeh Form, Thans yeu fsr your rn 
hsiping us fivfilyat* our •tf*€f ly«n«ii^ 



and axpiainvi, 

whM* at tti* Training Csntsp 
•KpiaiiiadT 

J« Training Mtariafi, pps^idad ta yau 



4, H€9 efta?t vlll tha aeni'anTs sf rhfi 
^aypsa aid y5U in dslng ysgr Jab 

Dveraiip i that tha contant at 
thm niitarlai prasantad In tnls eauria 
•ras _ 

a« Th« pad* af this asupsa «aii 



1, Tfia fnstPucter'i asnify ta gsnvay 
th»ir ^aviadga at tha lubja^t sai 



f« stt^ «as tha initpuat^ 

vai i<^prapapad7 

3s Nsw attafi ware Tha praiantat I ans 
elMF and In a lag leal srdar? 

4* Tha tfifcTpuGTor an^fiyragad laapnlng 
and wii a SBTlvatop , 



5, Tha aeportuftity te Intaraat *iTh 
tha Initpyqter «ai 



is Tha InitrustSF was as I a ts 

ipastftaaily raiata tna training 
ts mf Jq6 _ _* 

7, Tha inatrudtsp iiai attaatlva In 
maatlng tnm asjrsa sbjagtlyas 





4 


3 


2 


1 


Always 




Saaat I Biea 




Navar 


I 


4 


3 


2 


f 


Thsreyghly 




AdaqyaTai y 




^espiy 


5 


4 


3 


2 


1 


Es+r»wly 
Nal pty I 




iuf f Iciant 




i ndaguaTa 


5 


4 


3 


2 


1 


Frsqyanr iy 




Ottan 




Raraiy 


5 


4 


3 


2 


f 


Outsfsnd ing 




Satisfaatery 




in naad st 
Ravii ion 


5 


4 


3 


2 


I 


Apppapp lata 




laftHhat tS5 
Fas? {SlBwJ 




Fast (Slaw) 












5 


4 


3 


2 


f 


Outstand Ing 




SatistasTary 




Paop 


5 


4 


3 


2 


! 


Alssyi 




□ tran 




Raraiy 


5 


4 


3 


2 


I 


A j says 




Fraquant I y 




Rapaly 




4 


3 


1 


t 


Always 




Fraqyaht i y 




ftaraly 


5 


4 


3 


2 


! 


AlMiyt 
PrasaflT 




Qftan 




Rara iy 


5 


4 


3 


2 


i 


D^piataiy 




Saffiat imas 




Raraiy 


5 


4 


3 


2 


f 



Cofflp lata I y 
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Data I !_ 2/!?/eS 



COURSif 



'igure 4 



J WED! ATE COURSE FEeOBAQC 



POST TRAIN jftC FElDBAW 



PROGRAM REVfiW 



□ 
□ 
□ 
□ 



REFIR TO CH^TER 9 OF THE IDM FOR AN EXPLANATJOH OF THE ACTIONS REQUlRiD (P^AGRAPHS ffOlCATEBU ^Ci 
THE ACT j ON IS COMPLETES INITiAL* DATE AND ROUTE TO THE H&T INDiCATEO POSITION IN THE ROUTING PR^ESS. 



Jj#<EDl ATE COUMI 



POST TRAINING 



CDNDUGT 
SURVEy 



INST INIT/DATE 



RECEIVES/ 
DliTRU' TEI 



ASA iNiT/DATE 



CONDUCTS 
REVIEW 



TiPEA INlT/DATE 



RECEIVES/ 
DlSTRlBUTEi 



REVtiSi/ 
SUMMARIZES 
iVALUATES 



TS INiT/DATE 



9.2.3.2 
9.5.3.2 



REVIEwi/ 
RECO>«iENOi 



REVIEWS/ 
APPROVES 



INST, - INSTRUCTOR 

ASA - ADMINISTRATIVE STAFF ASSISTANT 

TS = TRAINING SUPiRV I SOR 

PTi - PRINCIPAL TRAINING SUPERVISOR 

D.H. ^ DEPARTMENT NEAO 

RRG - RECOTOS AND REPORTS GROUP 

TSPEA ^ TRAiNI^C SUPERVISE ^ PROORAH 
EVALUATION AND ASSESSMENT 



PTS INIT/DATE 



9.2.3,3 

9.3.3.3 
9.4,3.4 

9,5.3.3 



D.H, INIT/DATE 

9.2.3.4 
9.3.3.4 
9.4.3.1 
9.1.3.4 



WHEN APPROPRIATE 




MANAGER 

NTC 
iNFmEO 



D.H. INIT/DATE 



TSPEA INIT/DATE 

9.2.3. S 
9.3.3.5 
9.4.3.6 
9.5.3.5 



WHEN AFPROPRIATf 



PROJECT 
TRAO< I ftC 
SfSTEM 



RR6 
.2.4 
J. 4 
4.4 
.§,4 



INIT/DATI 



ATTAWMENT 8 



DATE I 
NT0-14Q 
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IMMEDIATE COURSE FEEDBACK PROCESS FLOW CHART 



Figure 5 



ACTION COLUMN 



' INFORMATION COLUMN 



CONDUCTS SURVEY 



INSTRUCTOR 



RiVlEWS 
EVALUATiS 
SUMMAniZES 



2 DAYS 



TRAINING SUPERVISOR 



10 DAYS 



REVIEWS AND 
RECOMMENDS 



REVIEWS AND 
APPROVES 



VERIFIES 
IMPLEMENTATION 



ACTION ITEM 
TRACKING 



PRINCIPAL 
TRAINING SUPERVISOR 



DEPARTMENT HEAD 0— 



TRNG. SUPV. FOR 
PROGRAM EVALUATION 
AND AUDIT 



PROJECT 
TRACKING 

SYSTEM 



H> MANAGER - N.T.C. 



RECORDS 



ATTACHMENT 4 DATE 12/20/BS 

REV. 9 

143DIA 
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POST TMlHXmi WW^WACK BURVmr 



Supervisor 



Data 



Area of Responsibility 

Trainee Name 

Course Name 



Course Number 
Course Date 



This form was designed to give you an opportunity to provide feedback information 
ooncerning the training your personnel received at the Nuclear Training Center, Thi 
comments that you afford may impact upon or redirect training. 



The trainee's knowledge of the job has improved s 



Comments % 



1 ^ not at all 

2 ~ adequately 

3 — significantly 



The trainee's ability to perform (skills) en the job has 
improved t 



Comments i 



1 = not at all 

2 = adequately 

3 ^ significantly 



3» The trainee's awareness of safety on the job has improved i 
Comments: 



1 - not at all 

2 — adequately 

3 ^ significantly 



I'he trainee's confidence on the job has improved: 



Comments : 



1 ~ not at all 

2 ^ adequately 

3 - significantly 



(over) 
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The trainee's understanding and follow through on 
chemistry procedures ha^ improved; 

Comments I 



1 = not at all 

2 = adequately 

3 ^ significantly 



The trainee's need for supervision has decreased; 
Comments t 



1 ^ not at all 

2 - adequately 

3 = significantly 



The trainee's ability to plan and organize has improved: 



1 ^ not at all 

2 = adequately 

3 ^ significantly 



please identify any obvious strengths or weaknesses not covered above* 



Please ratiirn coapleted form to 
y©ur training coordinator or to 
M# Parades, HTC, HC120. 
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POST TRMMIHG FmJBA^ FORM Figure 6b 

Employea Name Date 

(optional) 

bourse Title Course No. 



rhis form you ara about to complete will give the. Nuclear Training Center important 

feedback about its training programs. This information will be used to revise and 
improve this program* 

please circle the appropriate number. If your cirole either 2 or 1 please cominent on 
^our specific concerns • 

5 4 3 2 1 

Good Satisfactory Poor 



I . The training program as compared to 
my job responsibilities wasi 

Specific comments I 




AS compared to typical job duties? 
the lab exercises werei 

Specific comments: 



i. The ability of the course to 

relate to the knowledges and skills 5 4 3 2 1 

needed for my job wast 

Specific comments i 



. when used/ the plant specific course 

material was; 5 4 3 

Specific Gommentsi 



(OVmi PLEASE) 



Return aOBplet^ £on to yout 
training aoordlnatQr or to 



DATE: 06/1 6/86 
REV . I 0 _ 

NTC-172 
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5 4 3 2 

Good Satisfactory 



leaee cirale the response most el^arly representing the appli 



CONTENT in the program you attended* 


WORKER 

1 , Lab techniques 5 4 3 2 1 
and procedures 

2* Plant systems 5 4 3 2 1 

3, Fire and safety 5 4 3 2 1 

Comments: 


APPRENTICE 

1 • cnemistry _? ** ^ t 
fundamentals 

2, Analytical . 5 4 3 2 1 

3* Radiation 5 4 3 2 1 
fundamentals 

4. Radiation 5 4 3 2 1 
detection 

5* Chemical 5 4 3 2 1 
controls 

6. Station training 5 4 3 2 1 
Comments 1 














TECHNICIAN 

1 5 Chemistry 5 4 3 2 1 
fund* review 

2. Radiation 5 4 J ^ i 
fund* review 

3, Chem. lab 5 4 3 2 1 
instrumentation 

4* Count room 5 4 3 2 1 
instrumanta tion 

5. Chemical aontrals 5 4 3 2 1 
and systems review 

6* Station training 5 4 3 2 1 

Comments 1 


CONTINUING TRAINING 
Content 5 4 3 2 1 

Comments i 
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POST TRAINING FEEDBACK PROCESS FLOW CHART 

Figure 

ACTION COLUMN INFORMATION COLUMN 



SENDS OUT SURVEY 



nECEIVEd AND 
FOnwARDS SURVEY 



ADMINISTRATIVi 
STAFF ASSISTANT 



90 ^ liO DAYS AFTER 
COURSE COMPLETION 



TRAINEE'S 
MANAGER 



COMPLETES 
SURVEY 



1ST FOLLOWUP m DAYS 
2N0 FOLLOWUP 60 DAYS 



TRAINEE 



RETURNS 
COMPLETED 
SURVEYS 



RECEIVES AND 
DISTRIBUTES 
SURVEY 



REVIEWS AND 
PREPARES RESPONSE 



REVIEWS AND 
RECOMMENDS 



REVIEWS AND 
APPROVES 



VERIFIES 
IMPLEMENTATION 



ACTION ITEM 
TRACKING 



TRAINiE*S 
SUPERVISOR 



TRAINEE'S MANASER 



ADMINISTRATIVE 
STAFF ASSISTANT 



TRAINING 

SUPERVISOR 



1 



10 DAYS 



PRINCIPAL 
TRAINING SUPERVISOR 



DEPARTMENT HEAD 



-J— > MANAGER - N,T,C, 



TRNG. SUPV. FOR 
PROSRAM EVALUATION 
AND AUDIT 


1 

1 


— r ^ — — ^ — 

I 
1 
I 
1 





PROJECT 
TRACKING 
SYSTEM 



RECORDS 



144DIA 



ATTACHMENT 5 



DATE ia/20/Si 
REV. 9 



V.B.B.20 



NUCLEAR TRAINING CENTER NTCO 74A 

Pf RFORMANCE EVALUATION OF TJkAINlNG EFFECTlViNlSS (PETE) DATE] 10/27/86 

Q_B_5£RVATlQN FORM REV.: 1 

Piqure 8 

'[ON: S = HC JOB CLASglFlCATlQN{S) - - = = 

E: / / __. OiSERVER(S): - - 

■CS): 

?K ORDER # / iOS # -IT^ ^- /DUTYAREA: _ 

JESCRlPTlON: 



iScRvATION/EVALUATlON ITEMS OUTSTANDING 


SATISFACTORY 


UNSATISFACTORY ^WHY^ 


Did ptrsonnel follow appfOvtd Proeedure(s)? 

a. PrQeedure #: AP-31 (Cleanliness) 

b. Proeedure 






Proeedure #i 




d- Proeedur^ #] 




e. Procedure #: 




f Procedure #r 








f iviiyiEiM 1 ^ r - - -- 


Did pifionnel use pfoptr tigging verification proctdurii? 
IMMINTS: . ._ — ^ — 












Were there delays m comrnencing job due to irnproper planning? 
a List: . 














DMMENTi: 






a. Were proper tools pri'Staged for the job? 

b. Was proper tool usage Observed? 

JMMENTS: 








Was cilibration verifieation observed? 

dMments' 









' DATE: 10/27/86 

REV.: 

PIT! ObitfvatiQn Form Page 2 



►BSERVATlON/iVALUATlON ITEMS OUTSTANDiNG 


SATiSFACTORY 


UNSATISFACTORY .\^.m)Y? 


Were ALARA cQncepti foliowed? 






J IV! I VI E !>! t 3 , ^ — ^ ^ — ^^.^ — ^ 


7. Wtre Safety Procedures fsHQwtd^ 






^UlylJVlEN 1 3. — , — ^ ^ ^^^^ H ! — = — - 


Were communiCationi bttween departments effective? 
SS: RO' 






HP: Securitvi 








}. QA/QC involverrient Qbsarved? 

RflERiTC . 










10. Did the job ptrformince indicata the vvofker undtritanding of 
basic theorical knowledge? 

-/^RjiltMEhiTe. 










Skill Observations 




)BSiRVATION/EVAHJATlON ITEMS OUTSTANDING 

1 1 During performanca of the job, what skills were observed? 

a. 


SATISFACTORY 


UNSATISFACTORY - WHY? 


b. 




c. 
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PtTE ObiervatiOn Form 



DATE: 10/17/86 
REV; _L_ 

Pip 3 

Figure 8 



S^gnibnt Change Recofnrti^n* ^ dations: 



^#pld Actions ^ 



To b# Complited By 



Approved ly: 



Date Coinplttid 



Training Dgpartrn#nt Head 



Datt^ 
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NUCLEAR TRAIN I NG CiNTER 
PERFORMANCE EVALUATION OF TRAINING EFFECTIVENESS 
"PETi" PROGRAIV! 
OPERATION OBSERVATION FQRtVI 



STATION: S-HC JOB CL^SSIFlCATlON(S): 
DATi: / / QBSERVER(S): _ 



TIME(S): . 



jTA#: 



JOB DESCRIPTION: 



_/ DUTY AREA: 





SAT 


linsat~ 


N/A 




UTILIZATION OF PROCEDURiS 












2, 


COMPLlANCi WITH TAGGING RULES 












3. 


JOi PLANNINC3 












4. 


SAFITY PRACTICES 












5. 


LOG KiiPING 












6. 


COMMUNICATIONS 












a. Fa€#-te'Fae# 








b. Remett 








7. ALARA APPUCATION(S) 








S. 


UNDIRSTANDING AND PERFORMANCi OF TASK(S) 












9. 


USE OF GOOD ENGINEIRING PRACTICES 












10. 


OVE^LL PfcRFORMANCE 













* REQUIRES COMMENT 
COMMENTSrrRAINING NEEDS 



DATI: 10/27/81- 
REV.; Q , 
fsJTM74l" 
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Figure 9 
PAGE 2 
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OF TRAINING EFFECTIVENESS 
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QIJESTIOKUAIRE 
MEASURlWij TRAINING'S IMPACT 
R.A* Corf ield 



FACTS I 

1. la your utility orgaiiip.atioii us^aiin any systeniatic, 

objective J quantitative method for Measuring Training's 
Impact? 

2* If so, can you describe what you are doing? 

3* Do you know of any utility using a systematic approach to 
Measuring Training's Impact? 

4, Do you know of any published results of hard data on this 
subject either within our industry or in a similar or 
related industry? 

5, Do you know if INFO is doing anything in this area? 
OPINION I 

1* Can we devise a systematic quantitative approach? 

2, If so, what would that approach be? 

3. How accurate might the results be? 

4* How much effort (or expense) do you think is involved? 

5, Will your utility (or the industry) financially support 
the development of such a systematic approach? 

6, What will the money buy them (us)? 

(What would be the value of the results in terms of 
increased, decreased, or redirected efforts or application 
of resources?) 

?• Should we even ask the questions if we are afraid of the 
answers? 

8. Should INFO be responsible to do this? 



IDEAS I 

1* What else should we be considering? 
2. Would you like tj be involved? 
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THE PAYBACK ON OUR TRAINING INVESTMENT 



William A, Nichols 



ABSTRACT 



A cramendous amount of resources have been invested in 
training over the past few years. The return on the 
training Investment will be evident over the next several 
years* The return will be In the form of financial savings 
to companies and improved perforinance of personnel* At the 
Cook Plant, we expect the return to outweigh the investment. 



The D. C* Cook Plant is a large^ two unit PWR that has been in 
operation since 1975* Following the TMI accident, we, along with 
the entire nuclear Industry, began reassessing how training was 
being done and how it could be improved to more effectively prepare 
personnel to perforni their jobs. At the Cook Plant, we faced the 
potential of major changes to existing training programs to make 
them truly performance based. The outcome of our effort is that the 
NLOj ROj SRO, STA, and Chemistry programs were accredited in October 
1986 and the remaining 5 programs will be evaluated by INFO during 
an Accreditation Vis^.t the week of May 4th, 1987. 

The large investment in training made at the Cook Plant is 
exactly that, an investment. As with any Investment that is made, 
the return is expected to be worth the investment. The investment 
that has been made is two-fold, human resources and financial 
resources. Since early 1984, our Training staff has grown from 20 
professionals, involved primarily in Operator and GET training, to 
today's 57, which includes all skills programs and staffing for our 
simulator. Along with the growth in the Training staff, 
approximately J 5 man-=years of subject matter expertise was used to 
supplement our Training staff in the development and revision of all 
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of our training progranig. The financial investment being made 
focuses arouTid an 85^000 ^£t^ training facility that will house a 
full scope simulator along^ with laboratory, shop and classrooms to 
support all of our tralnlteng programs. The financial invastment for 
the training facility and simulator is approKimately $30»000,000* 

Measurlni the paybacKk on this large Investment is difficult* 
Many of the areas in whictQi training is a contributor do not have a 
direct financial value as^^ociated with them. Many of the areas that 
do have a financial valu© associated with them are affected by many 
factorss of which trainin^g is only one* On January 11th of this 
year, a new consecutive d^^y run record was established for the Cook 
Plant. We subiiquently efcKtended that record to over 200 consecutive 
days. This is parclcular^3.y significant since we are an ice 
condenser plant which re^tmiires additional planned shutdowns to 
perform ice condinier Sur^waillances* The improved availability of 
the plant has a direct flt^ancial payback to the company. Training 
is a contributor to this ^ffactor, but how much of a contriburot 
cannot be meaeured. 

The savings to our ccscipany by investing in our own site 
specific simulatot will difficult to measure* The cost per year 

to our company to operate our own simulator should not change much 
from what we spend now t£> contract this training. What we do expect 
is to be able to provide i^iore quality time on our simulator since it 
will repllcata cur plant* We are looking at using our simulator for 
practicing startup and sjiite^tdown operations prior to performing these 
evolutions on the plant. This extra practice by the operating 
shifts following an ext^^ifled shutdown or a record run could save 
time on either end of a^ Q»utage* Each day's savings on an outages 
because we've been able CQ« degas or heat up mora efficiently as a 
result of the prictices ^1^11 save our company approximately 
$500 s 000* Saving one day aach year for 30 years will result in a 
financial savings to th^ c -ompany of $15,000*000, 
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Another area in which we expect a benefit from having our own 
simulator is in the; area of unplanned reactor trips. The Cook Plant 
is susceptible to reactor trips during startups resulting fron 
manual control of ateam generator levels. Controlling steam 
generator levels on a startup Is very difficult since we do not have 
bypass valves around our main feed regulating valves. During tha 
last SALP appraisal period, there were 3 reactor trips during 
startups when feedwater control was in manual. Recovery from this 
trip can take a day. We plan on using our simulator for extensive 
practice on controlling steam generator levels. We feel this 
practice will reduce the number of unplanned trips during startups. 
For each trip that Is prevented, we expect to save $500,000 a day. 
Saving one day a year from these trips, over 30 years again results 
in a $15,000,000 savings to our company. 

One of the programs that changed dramatically as a result of 
the accreditation process was our SRO program. Since the chaniei 
were made, our pass rate has Improved from approximately 75% to 921. 
The cost of putting an SRO candidate through this program is 
approximately $20,000. Improving the pass rate results In fewer 
repeats In the program at $20,000 each. We conduct the SRO prograin 
for about 12 candidates per year. Two additional passes per year 
applied over the next 30 years will result in a savings to our 
company of $1.2 million. This same logic can be applied to the RO 
program; however, at this time, data is not available since our 
first group going through our revised program will not take the 
license exam until August. The savings for the RO program could be 
much higher since it costs approximately $50,000 for each candldata. 
We expect the addition of our own site specific simulator to furthar 
improve our pass rates on license exams. 

A performance appraisal of each nuclear utility's operation Is 
conducted each year by the NRC. Training can have a large Impact on 
a plant's SALP rating, in the last SALP report of the Cook Plant, 
training was discussed directly in 8 of the 11 categories measurid. 
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Since training as a contribiitor to such a wide range of plant 
operationSi it stands to rea^ojithat Impc^oved training can lead C, © 
improved SALP ratings. The %^alustion q£ our plant's parf orraaii^e by 
the NRC and by INFO Is eKtreHifily lmpoirtati.t to our company. 

We expect returns on pt*t training Itxvastment in many othet' 
areas and will be evaluating th^ii over the coming year. We B^pB^ct 
that some tasks in all ptOgf^^soiay take lees time to parfortQ ag 
rasult of improved parf ormaiiQ^ basad training* This has an added 
benefit of reduced radlaClo^ exposures, "We expect to sae more 
consistency in the parf ofmaJiQ^ of tasks* Related to this will be a 
mora afflciant utilization at resourcas* 

A lot has bean Invested in trailing over the past few yaats* 
The financial savings to a aQtnpany result Ing from training Is JuS^^ 
one measure of the payback^ Cte payb^clca from training that atmtm 
not easily measured but ar© e^ctriinely linp ortant Includa incraaaed 
confldenca in job perf otmatiCt^ ihsping of attitudes * and Improved 
efficiency of operations. At thi Cook PI snt we feel tha payoff ftaffom 
training will more than justify the investment* 




INITIATIVES IN TRAINING PROGRA M EVALUATM OUTSIDfl ~"THE 
NUCLEAR UTILITY INDUSTRY 

C. James Allen 



ABSTRACT 



•J^a fining literature is reviewed, and program avaluative poractlces 
outside the nuclear utility industry are reported. The findiiiaas Indl- 



cate som^^s innovations in philosophy and practice of propram #\^aluation 
although not necessarily In the context of evaluation as a r^U^te to ' 
assesainas -he impact of training. Program evaluation Is destft — ±bed In 
the context of the impact of training, suggestlnt continued #f ^forts 
to accePa-C a mulcivarlate concept of individual and organlzattor*Tial per- 



INTRODUCTION 

Th^ philosophies and practices that define instructional Systems in 
the nucJ.a-=sar utility Industry has roots in the exparlences of o^her indus- 
tries at^da businesses, in the annals of educational researchers^ and in the 
theories and concepts of the disciplines of Iducatlon and PsyCtnology, 
among oeh.-ers. As the nuclear utility Industry evolves Its metl^odologles 
and phliot Sophies via symposia, industry groups, ragulatory ma^iclSates, and 
proceduE-a; 1 refinement. It is Invaluable to returti to those roePt-s, to "ex- 
■parid the a gene pool" of knowledge, practice, and somd aducaticjnj-al princi- 
ple. As »-the past six to eight years have wltneasad an explosj,o-sn in In- 
structtofl»l sophistication In the industry, other Industries #n*_d businesses 
have compiled many lessons learned. As econon'ic tides dictate t business 
necessities, training managerB in countless types of orRanlzaela ons strive 
to Justil^ and demonstrate training's contribution to the botCOtEm line. 

CentE^al to the notion of measuring training's impact is t'hmm more 
fundaineni:a«l concept of training program evaluation. While addressing 
.^ngact li*ip«lies assessing training's value to the o rganization , ^.a. to 
Che botttfiito line, training program evaluation encompasses asses«nEiient of 
training's., value to Individual trainees, departments, and othe* components 
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of the organization. Assumptions are made about relationships of those 
parts to the whole ^ Indeed ^ program evaluation remains somewhat a necaa-- 
sary but not sufficient mechanism in efforts to measure training's impact. 



Sara Steele described trends in program evaluation in 1973", Her 
observations seem yet to be realities in many evaluative endeavors: 

1) Program evaluation is a process rather than a product; it is more 
than a specific methodology Cestlmates are that more than 50 evalua-- 
tion methodologies evolved in the late 60 's and early 70 's). Yet ^ 
In order to maKimize the utility of evaluations It must serve as a 
means of forming judgments about programs and a means of collecting 
Information to use in comparing alternatives, 

2) Program evaluation is more than eKamining attainment of objectives* 

Objectives undoubtedly provide a focus for our attention, but training 

2 

also produces unexpected side effects, good and bad, Sadler describes 
"goal-free*- evaluation, in which objectives of a program are set aside 
while evaluation occurs; prespecified objectives might not be met at 
alls while other results appear that are too positive to ignore, 
Steele goes on to state that eKamining attainment of objectives is 
primarily a descriptive activity, while evaluation, insofar as value 
is its central tenet, addresses whether the results of training are 
important, whether training contributed more than other events that 
could have occurred, and whether results contributed to overall needs 
of the organisation, 

3) Program evaluation is more than evaluating program results, Summative 

(end-of-'program) evaluations are well established in most training 

organizations. Formative evaluations, performed while training is 

in process or even development , provides more inmiedlate feedback and 

3 

influences the current program as well as future ones. Komras sug^ 
gests conducting the first evaluation after completing the first draft 
of trairiing materials* The emphasis on evaluation, according to 
Steele, is shifting from, solely the summative to the relatednesa of 
sunmiative and formative. 



THE LITERATURE 
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4) Program evaluation is mora than inBtructional evaluation. Instruc- 
tional evaluation la concerned primarily with specific course or 
prograni activities. Program evaluation is concerned with the cumula- 
tive effects of a series of instructional events. Program evaluation 
focuses more on overall organizational needs than on specific objec- 
tives or individual states of learning. 

So the contemporary trends in program evaluation expand its role 
beyond assessment of training for the sake of the training system alone. 
Evaluative methodologies cannot be far behind. Kelley, Orgel. and Baer^ 
state, "The development of evaluation methods that identify a training 
program's benefits and coats accurately and thoroughly may become the 
most Important contribution of training and human resourcA development 
in the 1980 's" (p. 32). They advocate the statistical analysis of prs- 
and post-training performance measurements taken with trained and untrained 
(control) employees. Further, they recommend using graphic comparisons 
of performance data to Illustrate how great the differences in performance 
actually are. Indeed, statistical analyses Identify the results of 
training beyond what chance might produce. Finally, they espouse the 
collection of on-the-job performance data in lieu of only trainee and/or 
supervisory questionnaire data. This advice, although well-based In sound 
theory, is fraught with difficulties of implementation. Moat notably, 
the use of untrained control groups for statistical comparison continues 
to be a logistical and ethical near-Impossibility. 

Quinn and Karp^ also recommend the use of pre- and post-training 
assessment, but utilizing questionnaires that are validated (by, for 
example, correlation of alternate form questionnaire responsBs). 

The training, educational, and psychological literature does not 
abound with true Innovations In methodologies or practice In the program 
evaluation. Philosophies are evolving and will ultimately find expression 
in varying degrees. 
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IN OTHER INDUSTRIES 

Lessons are being learned in industries other than nuclaar utilities. 
Salingar^ surveyed evaluative practices in the Federal government and 
described several innovations. A program of the DepartTnent of Agriculture 
teaches evaluators both the techniques and orientation of evaluation* 
The Department of Education turned trainers into evaluators , assessing 
via interview the success of trainees^ application of learned skills 
some time after training* The U*S* General Accounting Office employed 
both formative (assessing trainee reactions and knowledge gained) and 
sunmaative (assessing application of learned skills on the job) evalua- 
tions = process and product evaluations ^ respectively* The Department 
of Labor compared productivity goals and actual achievements as a means 
of assessing course effectiveness* 

In a light manufacturing industry in Texas p evaluation of training 
of QC inspectors was conducted following an er.ght=week course* The 
evaluation deemed the program successful* The course was reduced to 
three weeks in duration | no evaluation was conducted again* The commit- 
ment to evaluation must be ongoing to assure that changes in the programs p 
the participants i the instructors, and/or the Drganization do not render 
initially positive evaluation results invalid* 

Likewise, an industrial construction firm evaluated training via 
Informal field testing, then abandoned evaluation efforts after few 
attempts. Neither employees nor trainers welcomed the evaluation, sug^ 
gesting the need to de--personalize such activities and to stress the 
positive aspects of evaluation* 

A major oil/gas refining Industry evaluated the success of a 
modular program for technical/production personnel. The program was 
designed on the basis of a needs assessment involving IslOO employees' 
responses to need-f or-training on 88-89 "competencies*" A statistical 
cluster analysis identified the content of modules* The first few modules 
were Implemented In a pilot program with experienced supervisors and 
managers* Pre-- and post-training questionnaires assessed which behaviors 
were affected by training. Performance Improvements were indicated. 
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There are numbers of fossil utilities adopting perf orniance-l.. 
qualification procedures which double as assessmenta of tralnlni etfectlv-e- 



ness, 



Training program evaluation is certainly but one aspect o£ Assessing 
the impact of training. Both the literature and Industry axperiw 
outside nuclear utilities support generalizations about cominon pi>oUeins 
to be solved in assessing the impact of training. Human behsviDt is 
affected by many factors j Job performance, only one category of betoior, 
can be explained partially by factors attributable to trainitiR. butonly 
partially. Second, organizational performance - and the bottoin line - 
is also affected by many variables, of which human performance is m om^ 

The attempts to assess the impact of training will impirovs auccesa 
Is achieved in identifying and controlling the intervening vart#bies in 
both human behavior and organizatlonai performance. 
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The following papar wag not received In time to be published In the pro- 
ceedings. Space Is provided below for notes. 



Training I Is It Really the Solution? 

Joseph A. Gonyeau, Northern states Power Company 



NOTES : 
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EVALiOATltl Q PERFORMANCE MEASURES TO DETBRMINB TRAININQ EFFECTIVENESS 

Dr. Robert W. Klemm 
Anthony S, Felza 



ABSTRACT 



This reBearch was conoolved and dedloatad to helping the CECo train- 
ing organization become a more Integrated part of the corporate buslneaB 
The target population for this atudy was nuclear and foBall generating 
Btatlon employeeB who directly Impacted the produeUlon of electricity 
The target sample (n-igo) Included i InBtrument, meciianical , and elec- 
trical maintenance personnel i control room operators; engineers radia- 
tion chemlstB, and other technical apeciallBts; and equipment operators 
and atterndants. A total of four InBtruments were utlllzad by thlB 
study. Three InstrumentB were administered to the generating station 
personnel. These included a demographic form, a laarning style profile 
and a motivational style profile. The focal Instrument, a performance ' 
skills rating form, was administered to superviaory personnel. Data 
analysis consisted of three major parts. Part one established Internal 
consistency through Gronbach alpha statlBtica. Part two provides summary 
statistica and breakdown tables for important variables. Part three 
provides Inferential statlBtics reaponding to the research questions. 
All six Performance Skills variables discriminated significantly between 
the "trained" and "non-trained" groups (p .001). in all cases, the 
mean value for the "trained" group exceeded the mean value for the "non- 
trained" group. Implications for further research Indicate that training 
does have a quantifiable effect on Job performance. 
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BACKGROUND 



The relationship batwean training and parformance has long baan an 
Isaue that has deflad objactiva rasaarch [1], CGjomonwaalth Edison 
(CECo), as well as othar utilitleB, have a naad to know "how" to relate 
training benefits with tha overall corporata mission i ganaratlng 
megawatts efficiantly and safely. 

An article in the PeDvuary 1987 issue of tha "Training and Development 
journal" bast axprassad the changing role of tralnars * This article 
statas J " . * * that Instead of training and davaloplng others in the 
time-honorad way* many more of you will ba Involvad In managing training 
battar, accounting for Ife better, and finding ways to do it better. 
You will, in short* be doing work that supports and advances a critical 
businass function C2]," 

This resaarch was conceived and dedicated to helping the CECo training 
organization become a more integrated part of the corporate business. 
It was hoped that the process of doing this research would heighten 
tha awareness of managemant and trainers to tha fact that the training 
function does, and will continue to, improve employee parformanca, and 
contribute in a significant way to overall corporate objectives. 



The target population for this study was nuclear and fossil generating 

station employees who directly Impacted the production of electricity. 
The target sample (n^lSO) Includadi instrument, mecha ileal, and elec^ 
trical maintenance personnel; control room operators; engineers, radi-^ 
ation chemists, and other technical specialists; and equipment operators 
and attendants* Training records from each participating station were 
used to identify employees who received more task^specif ic training 
than their counterparts. The quantity of training received was the 
criterion used to form the "trained" and "non-trained" research groups. 



METHODOLOGY 
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op ifiwaiy, leaa than 50$ of requlrsd couraeH completed verauo parsons 
c, ^ l«t ,g 80-10035 of couraea. ThlB method excluded the "annual -type" 
i ajiii"-;^ couraea. 

^t^, collection was conducted at each participating generating atation. 
IV J methods were used to yelec;j participants. Pirst, convenience aam= 
Piing Which included all available perBonnel. Theaa autijectB were asked 

their superviso2.s to complete the survey instruments. Subjects were 
randomly selected from the convenience sample, and then aasigned to 
either trained or non^tralned research groups baaed on training records. 
Second, specific sampling, which included participants identified by the 
researchers through training records. These subjects were randomly 
selected from training records prior to data collection. 

All participants were asked to complete a series of three questionnaires. 
The Immediate supervisor of each participant was asked to rate their 
subordinate using the pwformance akilla instrument. Inatruments re- 
lating to each participant, those completed by the employee and his 
respective aupervisor, were stapled together to form a packet. Each 
packet was coded either "trained group," or "non-trained group." All 
names and identifying markings were removed from these packets to main- 
tain anQnymity^^ 

Throughout this project, only the researchers knew the group status of 
each participant. Selection blasea [3], Hawthorne and "halo effects" 
[4], and other threats to Internal validity [5] were minimized by the 
randomized, single-blind, selection proceas [6]. 

INSTRUMENTATION 

A total of four inatruments were utilized by this study. Three Inatru- 
ments were administered to the generating station personnel. These 
instruments will be highlighted more specifically m a follow-up study 
of employee learning prof lies. These included a demographic form, a 
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leading style profile , and a fflotivational style profile. The focal 
instrument, a performance skills rating tormj was administered to 
supervisory personnel. This Instrument was designed to elicit super- 
visory perforoance ratings for each partlGipant, 

Th^ demographlG questionnaire included age categories, gender, current 
and past Job classifications, and years spent in current and past Job 
classifications , 



The learning style profile [7] was designed as a self -analysis tool for 
identifying four basic styles through which the mind receives and pro- 
cesses cognitive Information. These styles are Conrrete Random (CR), 
Concrete Sequential ICS), Abstract Random (AR) , and Abstract Sequential 
(AS), Each style is characterised by learning, environmental, and 
interactional preferences for the learner (see Table 1 below), Learn-^ 
ing style preferences Mong generating station personnel may provide 
valuable Insights for training departments. Internal consistency coef- 
ficients [8] ranged from 0*89 to 0,92 for each learning style. Test- 
retest coef f iclente ranged from 0,85 to 0,88 for each learning style. 
Construct validity coefficients ranged from 0,55 to 0,76. 



Table li Learning Style Themes 



St yle 
(US) 

(AR) 
(AS) 

(CR) 



Environment 

low tolerance 
for distraction 

high tolerance 

low tolerance 
for distraction 

stimulus -rich 



Relationship 
to In^itructor 

traditional 
subordinate 

"guide" role 

expert 

Instruction 
guide 



Relationship 
to Students 

ordered 



collegial 
minimal 

v^ied 



The motivating traits inetrument [9] was designed to identify motiva- 
tional preferences. Each item rapresents a different motivating need- 
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state do], internal conalatency coefflclentB for each of three motlva- 
fclonal cluBtera ranged from 0.71 to 0.79. CluBter one repreaented self- 
esteem needs couplad with aff Illative and self -protective needs. CluBter 
tm represented Job security and orderlineBa needs. Cluater three 
represented ambition and creativity needs coupled with respect for 
authority and i'sgulatlons . 

The performance skills questionnaire waa deaigned to elicit performance 
ratings of plant personnel from their immediate aupervisors. Six per- 
formance Skills themes were identified by subject-matter experts from 
the generating stations, Program Development, and literature review 
[11]. Theae performance themes are: 1), concentration and awarenesB 
cr hazards; 2), handling stress and preaBure in Job taska; 3), experience 
and background aklllsi reaourcefulneas and problem=solylng approaches 

to Job-related tasks; 5). responsibility for equipmen;. and procedures; 
and 6), manual dexterity. Five items ware written for each performance 
theme. The items were written to provide a difficulty range for each 
performance theme. For example, item n was the simplest skill for the 
category, item #5 was the most difficiilt skill for the category. A one 
(low) to ten (high) Llkert Bcale was chosen to rate each item. The 
one-to-ten acale conforms to traditional, "base ten" rating norma, and 
counteracts any response biases from the cuatomary one-to-five scale 
used by the Company. 

InteT'nal conalstenay coefflcienta ranged from 0.93 to 0.97 for each per- 
formance theme (aee Table 2). Theae high reliability estimates also aup- 
port content validity issues. That is, the items appear to be Inter- 
preted alike by both aubjeet-matter exports designing the inatrument, 
and station oupervisoro utilizing the Instrument . The Internal con- 
sifltency coefficient for the total Instrument was 0.74. This indicates 
that separate ouboeoreo representing each of the six performance themes 
is more appropriate than an overall, total acore. 
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Table 2: Performance Skills Reliability Coafflcients 



THEME 

Concentration/Hazards 
Streea/PresBure 
Experiance/Background 
Resourcefulneas/PrQblam-solving 
RasponBibllity for Equlpmant, etc. 
Manual Dexterity 

DATA ANALYSIS 
FB^t One 

Data analysis conaiated of three major parts* Part one eatabliahed 
Internal conaiatency through Oronbach alpha atatiatica [12]* Factor 
Malyaia of the Performance Skills and Motivational Traits instrumenta 
ware used to eatabliah fewer, manageable, item clusters , Thaae item 
clusters were interpreted and named according to the underlying thematic 
content* Principal components extraction with varlmax rotation was 
used for these arialyaea [13]- Factors with eigenvalues greater than 
1*00 were retained* Pwt two provided sunmary statistics and breakdown 
tables for Important variables. Pert three provides Inferential ata-- 
tiaticB responding to the rase^ch questions* Discriminant analysis 
[l^i determined differences in predictor variables between employees 
classified by the criterion v^iabla. 

Part Two 

Factor analysis [15] was performad on the ParforniMce Skills instriiment. 
The results indicated a one-factor instruments Factor 1 (eigenvalue - 
22*16), Factor i (eigenvalue ^ 1.34)* The one-factor solution was dis- 
carded for two reasons. First, factor analysis was more sensitive to 
global rating from supervisors* Supervisors were suspected of applying 
a global image of each p^ticipant toward the instrument. The one- 
factor aolution represents the one-factor rating system applied by the 



COEFFICIENT ALPHA 

0*9353 

0*9455 

0*9654 

0*9442 

0*9587 

0,9688 
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respondents. Second, the raliablllty analysis demonstrated that all 
six perfortnance skills themes were appropriate to use as separate 
measures • 



The Motivational Profile was aubjected to data reduotlon using principal 
components factor analysis with varlmax rotation, A three-factor solu^ 
tion was retained for interpretation. Factor one (eigenvalue ^ ^•63) 
represented self-esteem needs coupled with aff illative and self-protee- 
tlve needs. Factor two (eigenvalue = 2.58) represented Job security 
and orderliness needs. Factor three (eigenvalue ^ 1*96) represented 
ambition and Qreativity needs coupled with respect for authority mnd 
regulations. 



Table 3^ Motivational ThMes Reliability CoeffiGlents 

™EME COEFFICIENT ALPHA 

Factor One 0.7420 
Factor Two 0,7063 
Factor Three 0,6176 



Part Three 

A variety of descriptive statistics were calculated for the overall sam- 
ple, "trained" group, and "non-trained" group. Important variables 
and statistics will be highlighted. 

Of the 150 participant's, 76 represented the "trained" group, and 7^ 
represenaed the "non- brained" group. Males comprised 98$ of the par- 
ticipants. Job Classifications ware summ^lzed as follows? maintenance 
(52$), operators (8$), EO/EA (11$), health physicist (2$), all others 
(17$), and missing (10$). The average value for Years in Current Job 
was 5-1 ye^s. The OTedomlnant learning style was Concrete Sequential 
(7^$). 



The following statistics were computed for the "trained" status group. 
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The average age range was 30-39 years old (50%). The average value 
for Yews In Current Job was 4,6 years. The average score on the Per^ 
formanoe Skills form was 165 out of 300 possible points, 

A stepwise discriminant analysis prDgram [16] was computed to detect 
differencea between the "trained" and **non-tralned" groups from among the 
six Performance Skills VK-Iables, The Perforoance Skills variables 
must meet or exceed a Bignlficanee level of 0,05 ( ^0,05) to qualify 
as discriminating variables* A test for the appropriateness of multi-- 
variate nomallty wsa demonstrated by Box's M statistic [17], The value 
for M was 2.3120; not significant, indicating that the discriminant 
analysis was appropriate given these data. 

All six Performance Skills variables discriminated significantly between 
the "trained" and "non-trained" groups (p ,001). In all cases, the 
mean value for the "trained" group exceeded the mean value for t^m 
"non-trained" Eroup, A classification progrM was performed to predict 
ona's group membership (trained vs, non-trained) given the raw scores 
from all six Perfomance Skills themes. Training group status was 
correctly predicted in 78% of the cases. Non-training group status 
was correctly predicted in 72$ of the eases. (50% correct predictiona 
would be expected by random guessing, or chance). 

Specific results for each Perfomance Skill Theme are shown in the 
following table. The results ^e expressed in percentages. 



489 



VI. 5. 9 



Table 


Performance Skills 


Percentages and 




— — — 




— _ — 


^ 


Parf ormanc a 


Non-Trained 


Trained 


Perceritaj 


Skills 










Inaraas) 


1 , Awara/Ha^ards 


56.6 


TO A 


16.2^ 


2 ^ Strass/Prassuraa 




67 * 2 


13-4% 


3. Exparlenee 




72 .6 


15.651 


^ e Problaffi*"SQlving 


53*6 


67.6 




5 • Responsibility 


55-0 


69.4 


14.4$ 


6* Manual Dexterity 




70.4 


16. OX 


Overall 


55.1 


70.0 


14.9* 










Median Aga Level 


35.0 


35,0 




Years In Current 




5.7 




Jot 









SUroiAHY 

implications for further reaearch Indicate that training doeB kave a 
quantifiable effect on Job perfow.ance. The following queBtlonB =- 
"Were the results what one would expect?"; "la there enough Improvement? 
and. "How much are we aaving, or aa-e we Bpendlng too much?" can only 
be answered In the context of this study. 

Expectations were optlmlatlc, that 1b, we expected ponltlve reaults 
from our training. How much? That was not an issue given the newnesB 
of our "performance-baaed" training efforta over thr past two-and-one- 
half years. To better understand the queations of expectation and ade- 
quate improvement, one has to know where the standard, or baseline, 
exists, we now know whore CECo'a baseline exists, and subsequent im- 
provement measures may be developed from thla point. 

The queation of "bottom-line" dollars la an Implication that bullda 
upon the results of this reaearch. The next step, or "Phase Two" of the 
project, is to incorporate these data and methodology Into research 
designed to correlate human performance (as impacted by training) with 
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plant pai^fomanGe. The copraBponding correlations should yield bottom 
line data regarding coat-ef f ectiveneas and training progrma . 
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